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FOREWORD 


This  report  discusses  the  background  and  current  practices 
employed  in  the  development  and  evaluation  of  crew  systems  for  aerospace 
vehicles.  It  draws  upon  experiences  in  a  research  and  development  group 
of  the  Air  Force  for  background  and  details  the  steps  in  the  process  as 
they  are  currently  perceived. 

It  is  a  report  on  progress  made  in  applying  and  expanding  upon 
the  philosophy  presented  in  ASD  TR  61-545  (Ref:  1).  That  report, 

"Cockpit  Control-Display  Subsystem  Engineering",  was  based  upon  eight 
years  of  experience  and  proposed  a  method  and  a  management  philosophy 
for  developing  cockpits  as  integral  parts  of  weapon  systems.  This  is,  in 
effect,  a  final  report  by  the  authors  on  twenty  years  of  work  for  that 
proposal. 

A  prime  goal  underlying  that  philosophy  was  the  establishment  of 
effective  Air  Force  dominance  and  control  over  the  cockpit  and  in  turn 
over  the  system  and  system  effectiveness.  Recent  experiences  are  highly 
encouraging. 

The  development  of  a  methodology  for  crew  systems  has  been  a 
long  and  arduous  affair.  Significant  steps  have  advanced  the  capability 
during  the  years  but  the  total  advance  has  not  been  adequately  documented. 
In  general,  the  documentation  has  tended  to  address  specific  techniques 
or  facets.  It  is  the  intent  of  this  document  to  consider  the  cumu^tive 
experiences  and  practices  of  the  Crew  Systems  Development  Branch  (FIGR) 
and  its  predecessors  and  to  present  a  summary  of  a  practical  methodology 
for  crew  system  development  and  evaluation  in  its  present  state  of  develop¬ 
ment.  ~  - 

In  reviewing  reference  material,  an  interesting  observation  is  that 
reports  on  methodology  are  not  very  perishable.  Bibliographic  material  of 
twenty  years  ago  is  still  germaine.  A  note  in  a  1961  report  (Ref.  1)  said 
that  the^ nature  of  the  work  changes  with  the  passage  of  time  -  still 
true  -  but  the  methodological  extractions  remain  valid.  With  the  passage 
of  time  we  have  been  able  to  expand  the  detail  treatment. 

The  methodology  and  (management)  philosophy  employed  in  crew 
system  development  was  first  described  by  Kearns  and  Ritchie  in  1961  ' 

(Ref.  1).  That  approach  has  remained  as  the  theme  and  has  been  expanded 
upon  by  the  developments  that  have  ensued  and  by  the  many  engineers  and 
scientists  who  have  worked  in  or  for  the  branch.  Expansion  upon  this 
theme  has  led  to  a  comprehensive  approach  to  Control  Systems  and  the 
advocacy  of  a  Control  System  Science.  Expansion  beyond  the  scope  of 
control  systems,  while  a  viable  option,  has  not  yet  reached  a  satisfactory 
state  of  understanding  and  documentation. 
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In  reviewing  the  1961  report  (Ref.  1),  I  have  found  that  the 
changes  since  then  relate  to  generalizations  on  technical  groupings 
(technological  advances  have  given  us  flexibility  and  more  opportunities) , 
and  level  of  discussion  (the  experience  of  twenty  years  provided  a  wealth 
of  detail  for  filling  gaps  and  expanding  upon  concepts).  By  and  large, 
the  philosophical  thrust  has  remained  constant. 

Control- Display  as  a  uniquely  identifiable  work  area  within  the 
USAF  received  its  prime  impetus  in  the  early  50's.  It  has  progressed 
steadily  in  the  development  and  validation  of  techniques  for  advancing  the 
systems  considerations  as  a  means  to  providing  suitable  capability  for  the 
crews  of  USAF  aerospace  vehicles.  To  provide  for  continuity  and  to 
clearly  establish  the  basis  for  current  methods  some  review  of  the  pertinent 
history  is  provided.  Section  I  addresses  this  background. 

The  specific  process  is  discussed ,  as  an  overview  in  Section  II 
and  in  detail  in  Section  III. 

The  methodology,  as  now  perceived,  includes  many  of  the  ramifica¬ 
tions  not  ordinarily  described  in  a  technical  development.  Not  only  people 
skills,  but  organizational  interactions  can  profoundly  affect  the  course  of 
a  technical  effort  (Ref.  2).  In  a  system  of  significant  complexity  the 
effects  of  these  external  factors  are  of  such  magnitude  that  they  must  be 
considered  in  the  conduct  of  the  technical  effort.  This  concern  is 
addressed  in  Section  IV. 

While  it  is  not  the  intent  of  this  document  to  delineate  tests,  there 
are  some  aspects  which;  are  believed  sufficiently  significant  and  out  of  the 
ordinary ,  to  warrant  discussion.  This  discussion  is  contained  in  Section  V. 

This  documentation  was  prepared  by  ORLOC  under  USAF  Contract 
F33615-81C-3619.  Mr.  Richard  Moss  (AFWAL/FIGR)  was  the  Project 
Engineer.  The  report  was  authorized  under  Project/Task  2403. 

Special  acknowledgement  is  made  of  participation  by  Dr.  Malcolm  L. 
Ritchie  and  the  support  of  Greg  J.  Barbato. 
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SECTION  I 


INTRODUCTION 


In  the  original  concept  for  this  methodology  (Ref.  1)  the  following 
divisions  were  used: 

Preliminary  Investigation 

Preliminary  Design 

Research 

Development 

Test  and  Evaluation 

These  are  probably  as  good  as  any.  A  difficulty  in  discussing  the 
program  is  that  it  is  a  complete  entity,  and  any  attempt  to  partition  the 
program  produces  arbitrary  and  artificial  pieces.  Yet,  some  such  process 
as  partitioning  is  necessary  for  purposes  of  discussing  details  of  the  work. 

I  hope  that  the  readers  and  discussants  will  retain  the  thought  that  the 
pieces  must  be  viewed  in  the  context  of  ONE  program.  The  proverbial 
story  of  the  blind  men  examining  an  elephant  comes  to  mind  when  attempting 
to  discuss  a  weapon  system  -  we  can  talk  of  legs,  tail,  trunk,  torso  as  did 
the  blind  men  but  those  items  exist  only  as  part  of  the  whole  and  have  no 
meaningful  existence  as  independent  parts. 

Our  interest,  initially,  was  centered  on  the  development  of  the 
cockpit.  Our  goal,  as  stated  earlier,  was  to  be  able  to  present  requirements 
with  sufficient  specifity,  clarity  and  scope  as  to  assure  the  Air  Force  that 
the  final  design  would  be  suitable  for  the  job  and  satisfactory  to  the  crews. 
Not  clearly  stated  but  implicit  in  our  effort  was  the  intent  to  be  able  to 
measure  proposed  designs  so  as  to  determine  the  degree  of  compliance  with 
specifications.  The  first  major  effort  (Ref.  4)  (Ref.  5)  underscored  two 
important  points.  (1)  Even  a  small  effort  at  improving  the  design  in  a 
system  context  could  produce  impressive  results.  (2)  The  cockpit  relation¬ 
ship  to  all  other  parts  of  the  weapon  system  was  such  that  any  change  in 
the  cockpit  was  reflected  elsewhere  and  any  change  in  the  weapons  system 
impacted  upon  the  cockpit. 

This  second  point  may  seem  obvious  but  at  that  time  period  even 
adjacent  instruments  were  treated  as  independent  entities. 


THE  START 

A  common  concern  in  the  early  1950's,  was  the  congestion  on  the 
instrument  panel  and  the  lack  of  space  for  adding  new  devices.  New  systems 
clamored  for  space.  One  of  these.  Data  Link,  could  provide  command 
information  to  the  pilot  for  effecting  intercept  missions.  There  was  no 
ready  answer  on  how  to  add  it  to  the  panel. 
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The  "Whole  Panel  Program"  (Ref.  5,  6,  7)  was  initiated  to 
address  this  issue.  It  was  intentionally  limited  to  a  scope  which  was 
believed  to  be  within  our  capability  to  address  effectively.  As 
Mr.  Knemeyer  put  it,  "Don't  bite  off  more  than  we  can  chew."  The 
intention  was  to  produce  improvements  which  could  be  immediately  realized 
while  simultaneously  providing  insight  into  how  to  deal  with  the  broader 
aspects  yet  to  come.  This  first  step  produced  substantial  benefits  for 
the  Air  Force.  One  product,  the  "T"  scan,  became  the  new  standard  for 
Air  Force  instrument  panels,  replacing  the  "Sacred  Six"  and  the  "Basic 
Eight"  arrangements. 


Figure  1.  The  Whole  Panel 

The  Flight  Director  became  de  rigueur  for  all  subsequent  panel 
proposals.  Vertical  scale  displays  became  very  modish.  None  of  this 
came  easily,  it  was  a  long,  difficult  and  expensive  effort.  The  resultant 
design  appeared  in  the  F-102,  F-106,  and  the  F-105  as  the  Air  Force 
Advanced  Instrument  System.  Variations  have  been  used  in  all  subse¬ 
quent  Air  Force  aircraft. 
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Figure  2.  The  F-10  2  Cockpit 


IMPLICATIONS  DISCOVERED 

Several  key  issues  were  identified.  They  included:  display 
development:  interpersonal  relations;  technical  development;  and  systems 
integration. 

Display  Development:  A  systematic  approach  to  satisfy  the 
pilot's  needs  for  display  was  the  key  theme.  (The  pilot  oriented  approach 
has  continued  to  be  a  dominant  theme  for  the  display  problem  from  the 
viewpoint  of  the  pilot.  Several  things  were  evident.  There  were,  seemingly, 
conflicting  signals  being  given  from  various  instruments.  Interpretation 
of  an  overall  picture  required  considerable  mental  manipulation  of  the 
various  scales  and  dimensions  used.  There  was  a  need  to  harmonize  dis¬ 
plays  from  one  instrument  to  another  and  to  make  them  as  useful  as 
possible.  In  our  concern  to  improve  the  cockpit  in  terms  of  the  pilot 
viewpoint,  we  solicited  the  expertise  of  disciplines  skilled  in  determining 
the  optimal  configuration  for  use  by  humans.  Engineering  psychologists 
were  asked  to  participate  in  the  design  process.  This  request  set  precedent 
that  has  produced  an  abundance  of  benefits.  The  inclusion  of  this  specialty 
area  into  the  design  process  from  the  initiation  of  a  project  has  been  a  key 
factor  in  our  successes. 

Interpersonal  Relations:  Interpersonal  relations  among  members 
of  the  development  community  turned  out  to  be  a  significant  issue.  Each 
component  was  the  sole  domain  of  a  champion  (an  expert,  a  czar).  The 
attempt  at  harmonization  between  instruments  required  that  each  czar  re¬ 
linquish  some  of  his  authority,  some  of  his  domain.  There  were  con¬ 
scientious  concerns  expressed  by  these  people.  They  had  no  background 
data  to  support  the  contentions.  It  appeared  to  conflict  with  items  which 
were  then  of  great  concern  (GFE,  standardization,  mass  production,  etc.) 
The  probability  of  success  for  such  a  significant  change  appeared  to  them 
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to  be  slim.  Their  technical  world  pivoted  around  their  specialty  and 
the  idea  that  a  "system"  engineer  could  give  their  area  (as  well  as  many 
others)  adequate  understanding  and  consideration  was  foreign  to  their 
thinking.  They  required  proof  that  the  result  would  be  beneficial. 

(With  the  need  for  such  proof,  improvement  came  slowly.) 

The  engineers  reacted  to  an  inferred  criticism  when  experimental 
psychologists  were  employed  to  test  displays  and  controls.  Human 
engineer  became  the  label  for  these  people  and  developed  into  a  popular 
buzz  word. 

Actual  design  involves  compromises.  The  field  of  "Experimental 
Psychology"  was  deeply  involved  in  research  on  principles  and  the  idea  of 
some  scientists  participating  in  compromises  was  distasteful  to  the  purists 
of  the  field.  So  the  psychologists  also  experienced  stress. 

Technical  Development:  In  the  technical  area  there  seemed  to  be 
challenges  on  several  frontiers.  Servomechanisms  were  not  new  but  they 
received  considerable  impetus  in  this  program.  Requirements  conflicted 
with  restrictions  in  that  much  had  to  be  done  in  a  small  space.  Density 
(space  occupied  by  components)  in  the  vertical  instruments  reached  90%. 

For  the  first  time,  very  expensive  instrument  models  were  used  to  develop 
acceptable  lighting.  This  was  a  consequence  of  the  sophisticated  displays, 
the  large  faces  and  the  concern  for  a  balance  across  all  of  the  instruments. 
The  innovative  technique  of  wedge  lighting  went  into  production  from  this 
program. 

System  Integration:  System  integration  was  employed  in  the 
linking  and  switching  of  signals  from  black  boxes  to  displays  and  revealed 
many  benefits  to  be  derived  from  this  idea.  It  too  became  a  buzz  word. 
Interfacing  black  boxes  was  a  systems  engineering  task.  But  organizations 
were  concerned  about  jurisdiction. 

The  interaction  of  the  cockpit  with  all  parts  of  the  system  has 
significant  implications  on  Weapon  System  design.  In  fact,  it  is  a  unique 
relationship.  No  other  component  or  subsystem  has  such  a  relationship. 

It  affects  or  is  affected  by  every  other  facet  of  the  Weapon  System  by 
virtue  of  the  need  to  exercise  jurisdiction  over  the  entire  weapon  system. 
The  cockpit  is  an  intimate  part  of  the  weapon  system  -  not  an  adjunct  to 
provide  a  place  for  the  pilot. 

We  ultimately  developed  the  label  "Crew  System,"  meaning  this 
intimacy  of  cause  and  effect  between  the  cockpit /crew  and  all  other  ele¬ 
ments  of  the  weapon  system.  This  "crew  system"  is,  in  some  respects, 
a  phahtom  system.  It  is  not  independently  designed  and  built.  It  is 
really  a  way  of  looking  at  some  aspects  and  functions  of  each  of  the  other 
parts  as  they  relate  to  the  functioning  of  the  crew  in  accomplishing  the 
mission.  It  is  a  dependent  system.  The  functioning  of  the  crew  is 
affected  by  design  changes  in  a  subsystem.  If  each  subsystem  pursues 
its  own  course,  then  the  crew  functioning  is  subject  to  unpredictable 
changes.  Not  a  very  healthy  condition  from  the  standpoint  of  concern 
for  mission  effectiveness.  ' 
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The  crew  system  came  to  be  viewed  by  us  as  comprised  of  the 
crew,  the  crew  compartments,  the  instruments ,  the  controls,  the  phantom 
structure  just  mentioned,  and  the  gestalt  functioning  of  the  weapon  system. 
The  problem  was  to  bring  about  an  efficient  design  for  mission  accomplish¬ 
ment.  As  we  saw  it,  improvements  in  individual  displays  were  necessary 
and  could  be  pursued  immediately.  Improvements  in  crew  systems  were  a 
far  different  matter.  In  the  crew  system  issue  it  is  necessary  to  con¬ 
sider  relationships  with  the  whole  gamut  of  developers  involved  in  a  weapon 
system  program. 

A  very  difficult  challenge  was  the  coordination  needed  across  all 
design  efforts,  in  order  to  realize  the  benefits  of  systems  integration. 

In  the  design  and  development  of  a  weapon  system,  literally  thousands  of 
engineers  and  scientists  are  engaged.  Each  of  them  are  addressing  fairly 
specific  problems  with  defined  boundaries.  These  specifics  are  the  "legs," 
"trunk,"  "tail,"  etc.  of  the  system.  The  very  magnitude  of  numbers  (and 
independence  of  spirit)  militate  against  adequate  control.  (I  don't  mean 
to  imply  resistance  by  the  people.  These  are  people  with  integrity  and 
competence.  They  do  the  best  job  they  can  -  as  they  see  it.) 

It  was  considered,  therefore,  that  a  logical  method  to  gain  the 
effects  of  control  is  to  be  found  in  the  structuring  of  the  problems.  If 
a  scientist  is  presented  with  a  problem,  properly  structured  in  the  system 
context  and  adequately  defined,  then  the  solution  will  fit  the  system  and 
can  become  the  system. 

SCOPING  (Where  to  qo) 

A  singular  reaction  to  the  Whole  Panel  effort  was  the  oft  repeated 
questions  "Why  didn't  you  go  further?  Why  stop  at  just  the  flight  instru¬ 
ments?"  We,  of  course,  had  restricted  ourselves  to  what  we  considered 
to  be  within  our  capability.  We  were  thoroughly  steeped  in  flight  control 
and  flight  instruments  -  so  much  so  that  we  did  not  need  to  formally 
document  the  problem  which  our  "system"  was  addressing. 

If  we  were  to  expand  then  we  would  have  to  define  a  new  scope 
and  develop  an  acceptable  logic  for  the  newly  defined  boundaries.  We  had 
become  very  conscious  of  the  need  to  explicitly  identify  and  consider  every 
conceivable  detail  of  the  problem  and  the  system.  Expanding  beyond  the 
areas  of  our  immediate  expertise  required  considerable  care. 

A  really  difficult  thing  was  to  expand  our  concerns  across  the 
system  and  yet  not  encroach  or  threaten  the  other  system  designers. 

Over  the  years  we  have  expended  substantial  effort  in  under¬ 
standing  a  problem  only  to  find  that  we  had  restricted  our  scope  of 
interest  too  much.  In  successive  programs  we  expanded  the  scope  of  our 
initial  problem  definition  (from  flight  instruments  to  Whole  Panel  to  crew 
system  to  mission)  and  put  increasing  emphasis  upon  understanding  (or 
relating)  that  even  the  smallest  element  is  influenced  by  the  broad  system 


objectives  and  constraints.  The  challenge  was  -  where  to  draw  the 
line  -  how  to  establish  the  necessary  information  without  steering  each 
engineer  into  broad  mission  analysis  for  which  he  was  neither  equipped 
nor  interested. 

(By  no  means  did  we  start  out  to  address  the  whole  scope  of  this 
ambitious  idea.  We  puttered  along,  unaware  of  the  magnitude  of  the  under¬ 
taking,  a  fortunate  happening.  If  we  had  known  the  true  extent  we  might 
have  been  reluctant  to  start.) 

It  became  clear  that  mission  objectives,  mission  effectiveness  are 
the  goals  and  the  constraints.  We  have  found  that  as  we  understood  the 
application  better,  we  did  a  better  job.  .  .  No  ...  let  me  rephrase  that. 
As  our  understanding  of  the  application  improved  then  our  ability  to  con¬ 
vey  criteria  (needed  characteristics  of  performance)  to  vendors  and  to 
subcontractors  became  more  satisfactory  (to  them  and  to  us)  and  the  resul¬ 
tant  system  became  better. 

If  we  could  understand  the  crew /system  interface 
well  enough  and  could  relate  it  to  mission,  then  perhaps 
we  could  influence  the  problem  description  such  that  the 
factors  affecting  the  crew  systems  would  be  adequately 
expressed  as  criteria. 

We  haven't  satisfied  ourselves  as  to  the  complete  answer  as  yet  but 
we  feel  elated  that  we  have  made  much  progress. 

DEFINE  THE  PROBLEM 

We  originally  defined  the  starting  point  for  our  methodology  as 
the  point  when  a  general  mission  concept  is  defined.  The  label  used  was 
PRELIMINARY  INVESTIGATION  with  the  objective  being,  to  understand 
what  is  required.  What  we  did  was  to  start  talking  to  pilots  about  our 
ideas  and  their  perception  of  their  job  and  needs. 

There  has  always  been  a  transfer  of  information  between  a  pilot 
and  a  designer.  What  we  were  after  was  to  achieve  an  improved  under¬ 
standing. 

We  had  found  that  all  too  often  our  limited  appreciation  of  oper¬ 
ational  details  resulted  in  not  even  knowing  what  question  to  ask  of  the 
crew.  As  we  improved  we  then  found  that  the  crew  could  not  always 
answer  the  question  without  much  introspection.  Often  they  had  to 
consider  the  question  while  going  through  a  mission  before  coming  up  with 
the  answers. 
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It  is  not  enough  to  say  "We  want  one  degree 
sensitivity  in  pitch."  The  designer  must  understand 
why  the  pilot  told  him  that  -  in  what  context,  for  what 
purpose,  under  what  conditions,  and  a  thousand  other 
questions. 
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!"  Semantics  and  convention  were  fearful  enemies  of  communication, 

f  Initially  many  people  did  not  see  any  significant  difference  between  the 

two  following  statements:  "I  need  an  attitude  indicator,"  and  "I  need  to 
know  the  attitude  of  my  aircraft."  We  reached  the  point  where  in  this 
example  the  statement  changed  to  - 

"I  need  to  control  the  flight  path  of  my  aircraft, 
therefore  I  need  to  know  and  control  the  flight  path  vector. 

Pitch,  roll,  yaw  and  throttle  are  the  only  means  of  control 
which  I  know  are  available.  I  can  operate  with  displays 
which  show  altitude,  rate  of  climb,  speed,  heading,  pitch, 
roll  and  yaw.  Possibly  I  can  do  better  with  a  direct  display 
of  flight  path  vector.  "  (A  pitch  trim  is  a  crude  substitute 
for  flight  path  angle  which  is  sometimes  useful  over  a  limited 
range. ) 

A  natural  development  out  of  this  experience  was  to  have  our  lead 
people  join  with  the  crew  in  an  effort  to  educate  one  another  and  to 
mutually  define,  in  great  detail,  the  performance  required  in  order  to 
complete  a  mission  and  to  identify  all  of  the  ramifications  that  influence 
or  constrain  that  performance. 

It  was  very  satisfying  when  new  crews,  in  talking  to  our  designers, 
concluded  that  we  were  experienced  crew  members.  That  was  a  milestone 
indicator  of  the  effectiveness  of  the  communication  effort. 

As  this  aspect  evolved,  some  ordering  of  the  process  took  place  - 
for  transfer  to  other  teams  and  other  programs  -  for  organizing  data  ac¬ 
cording  to  subsystem  needs  -  for  assurance  of  continuity.  This  included 
documenting  several  iterations  expanding  upon  the  Statement  of  Need  (SON). 
The  resultant  description  was  organized  according  to  the  needs  of  the 
engineering  community,  contained  considerably  more  detail  than  the  original 
SON  and  was  consistent  with  the  desires  of  the  customer.  As  a  companion 
to  the  "Need"  statement,  a  description  of  how  the  vehicle  would  be  used 
was  developed.  This  was  a  narrative,  from  the  crew  perspective,  describing 
the  manner  in  which  the  crew  expected  to  use  the  weapon  system  in  the 
performance  of  the  various  missions. 

The  Preliminary  Investigation  (Problem  Definition)  has  reached  a 
level  of  maturation  that  is  very  satisfying.  That  is  not  to  say  that  it  is 
complete,  only  that  there  is  no  question  as  to  its  workability  and  value. 


SOLVE 


Conventionally,  when  the  design  starts,  many  teams  press  forward 
in  the  area  of  their  expertise.  They  have  received  their  marching  orders 
and  set  out  to  build  their  thing  (wheel,  brake,  engine,  airframe,  fuel  cell) 
and  they  are  prepared  to  build  the  best  they  can  as.  judged  by  their  peers 
in  their  technology.  If  they  can  meet  the  stated  requirement  (fit  the  space, 
meet  the  weight,  come  in  within  cost),  then  they've  done  a  good  job.  Un¬ 
fortunately,  that  does  not  insure  a  good  system,  at  least  it  hasn't  worked 
out  that  way. 
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If  the  resultant  weapon  system  falls  short  of  hoped  for  capability, 
it  is  hard  to  identify  one  of  the  subsystem  designs  as  THE  primary  cause. 
The  subsystem  designers  did  what  was  asked  and  they  can  prove  it. 

They  have  batteries  of  tests  which  are  used  in  the  design  process  of  each 
specialty  area.  As  the  designer  proceeds,  he  has  a  means  of  measuring 
progress  and  of  measuring  compliance.  If  the  design  starts  to  deviate 
from  desired  conditions,  it  can  be  readily  detected  and  put  back  on  course 
before  it  develops  into  a  catastrophe. 

It  appeared  to  us  that  two  points  were  of  concern:  (One)  There 
had  not  been  any  adequate  means  of  testing  the  functioning  of  the  crew 
system  during  the  design  process  (comparable  to  the  test  batteries  used 
by  the  subsystem  designers)  and  (Two)  there  were  no  meaningful  criteria 
to  be  used  in  a  system  evaluation  if  the  means  were  available. 

Our  efforts  in  terms  of  the  testing  involved  the  use  of  simulators. 
This  allowed  us  to  represent  an  entire  cockpit  while  measuring  performance 
on  a  specific  display  or  control.  The  use  of  pilots  as  subjects  improved 
upon  the  value  of  the  subjective  opinions.  For  any  given  device  it  was 
possible  to  load  the  pilot  down  with  a  flying  task  which  exercised  that  par¬ 
ticular  device.  Concentration  was  upon  improved  experimental  techniques 
and  that  absorbed  a  considerable  amount  of  time  and  resources.  In  the 
earlier  years,  at  least,  we  felt  that  we  had  made  great  progress. 

The  second  issue  did  not  seem  to  respond  as  well  or  as  rapidly.  It 
would  be  highly  desirable  for  the  crew  system  group  to  be  able  to  specify 
subsystem  and  component  functioning  in  terms  of  their  crew  system  function 
and  to  be  able  to  assess  the  impact  of  the  myriad  of  component  variations 
upon  system  performance.  The  small  daily  changes  among  many  efforts  can 
accumulate  to  significant  variation  in  system  performance  -  a  sort  of  gestalt 
of  perturbations. 

What  appeared  to  be  needed  was  a  more  thorough  determination 
of  the  requirements  for  each  item  with  emphasis  upon  delineation  of  the 
crew  system  aspects.  Also  needed  was  a  means  for  continuing  assessment 
of  the  collective  aspects  of  these  items  in  their  system  context  and  mission 
applicability. 

Delineation  of  the  requirements  turned  out  to  be  a  two  pronged 
effort.  One  was  concerned  with  supplying  a  greater  level  of  details  to  be 
used  by  the  individual  designers.  The  other  was  a  need  for  comprehensive 
requirements  descriptive  of  the  total  collection  taken  as  a  system.  (The 
Design  Concept) 

Assessment  of  the  collective  aspects  of  the  pieces  while  functioning 
as  a  system  was  a  real  puzzlement.  If  we  were  to  measure  how  the  collec¬ 
tive  pieces  functioned  when  employed  as  a  system,  then  we  had  to  de¬ 
scribe  that  system  functioning.  The  desired  functioning  was  one  or  more 
missions. 

Design  Nattative. — In  our  efforts  to  understand  the  problem,  we 
had  drafted  narratives  of  how  the  crew  expected  to  use  the  vehicle.  When 
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we  iterated  this  to  describe  how  the  vehicle  must  function  in  order  to 
be  responsive  to  the  crew  expectations  we  had  the  start  of  system 
oriented  criteria  for  crew  systems.  Our  "criteria"  for  system  perfor¬ 
mance  was  a  detailed  description  of  the  system  behavior  in  performance 
of  the  mission.  This  description  was  called  our  Design  Scenario  and  it 
became  the  bench  mark  for  assessing  capabilities  of  system  designs. 

Design  Concept.  —  The  key  players  in  this  crew  system  activity 
(which  we  began  to  call  "core  group")  devised  overall  system  concepts 
(including  features  beyond  their  ken) .  These  concepts  were  based  upon 
their  appreciation  for  the  effective  use  of  the  human  in  combination  with 
the  technical  capabilities  afforded  by  the  laboratory  efforts.  It  required 
a  broad  based  awareness  of  advances  in  physical  technology  and  a  good 
appreciation  of  human  behavior  and  physiological  factors.  The  system 
concept  produced  was  a  straw  man  device.  It  provided  a  basis  for  dis¬ 
cussions  with  the  many  specialist  organizations  and  for  a  continuing  dialogue 
with  requirements  people  and  crews  from  the  ultimate  using  command  (the 
customer).  As  discussions  progressed,  the  concept  was  modified  to 
respond  to  the  expertise  of  the  consultants  but  always  retained  the  crew 
system  concerns.  Tact  and  skill  in  human  relations  is  an  important  in¬ 
gredient  for  the  core  group  members. 

With  this  approach  a  general  (or  functional)  description  of  a  viable 
system  emerged.  It  did  not  define  hardware  but  included  descriptions  of 
the  capabilities,  functions  and  performance  that  many  experts  had  testified 
as  practical  (based  upon  existing  hardware,  state-of-the-art  for  their 
field,  or  extrapolation  of  current  R&D).  The  conceived  system  was  at 
a  practical  forefront  of  technology. 

(This  is  comparable  to  the  concept  of  architecture  for  a  computer 
design  -  in  this  instance  the  architectural  design  for  the  total  system.) 

It  was  in  this  stage  that  a  partitioning  evolved.  Although  not 
constrained  to  traditional  divisions,  it  frequently  reaffirmed  them  (e.g., 
communications,  navigation)  but  modified  them  according  to  system  oriented 
considerations.  Consequently,  specialized  teams  could  be  formed  and 
address  their  problems,  not  as  independent  entities  within  the  system, 
but  as  essential  features  of  one  system  where  the  interplay  among  the 
subsystems  is  as  significant  in  influencing  their  design  as  is  the  state  of 
their  particular  technology. 

Any  design  process,  as  it  progresses,  involves  compromises.  This 
usually  means  that  you  can't  realize  a  design  objective  as  you  had  expected 
because  of  a  factor  which  you  had  not  previously  known.  Therefore,  the 
design  becomes  different  in  some  respect.  The  designer  looks  at  alterna¬ 
tives  and  selects  based  upon  his  judgment.  This  judgment  relies  heavily 
upon  his  background  experience  (normally  his  field  of  specialization).  It 
was  our  thought  that  the  designer  have  sufficient  information  about  the 
system  (and  an  efficient  communication  with  the  core  team)  such  that  the 
selection  of  alternatives  include  more  awareness  of  the  system  implications. 


(It  is  details  or  lack  of  details  that  can  kill  you  and  we  wanted  to  improve 
upon  the  designer's  consciousness  of  the  details  that  were  system  related 
and  mission  related  and  therefore  would  react  to  his  compromises.) 

For  the  cockpit  it  is  conceivable  that  every  instrument,  every 
device  is  provided  by  a  separate  contractor.  Therefore,  the  effect  of 
compromises  in  each  design  could  have  (and  has  had)  a  devastating  effect 
upon  system  capability.  Control  or  approval  of  each  such  change  by  a 
central  authority  would  cause  a  tremendous  burden  and  delay  to  the  program. 
Awareness  of  the  systems  implications  by  the  individual  designers  appears 
to  be  the  only  efficient  and  effective  method  for  achieving  reasonable 
(tolerable)  compromises. 

Design  Scenario. — A  mockup  of  a  proposed  cockpit  design  is  not 
uncommon.  Our  variation  was  to  use  it  (1)  earlier  than  is  conventionally 
done,  as  a  means  to  assist  in  defining  details  of  needed  function  and  per¬ 
formance,  and  (2)  for  assessing  the  aggregate  of  these  specified  criteria 
in  a  systems  context  in  the  performance  of  the  missions. 

As  ideas  for  new  displays  developed,  particularly  those  that  were 
appreciatively  different  or  involved,  they  were  mounted  in  cockpit  mockups. 

It  was  a  lot  easier  to  explain  their  functioning  when  they  were  viewed  in 
the  context  of  the  crew  station.  In  explaining  the  more  complex  ideas,  it 
was  necessary  to  discuss  the  role  and  problem  at  issue.  The  "Design 
Narrative"  provided  the  base.  This  seemed  naturally  to  lead  to  role  playing 
demonstrations  in  the  mockup  which  were  showing  the  explanation  in  terms 
of  the  "Design  Narrative."  For  limited  problems  (part  task/part  mission) 
these  informal  discussions  sufficed.  With  expansion  in  scope  and  in  the 
numbers  of  subjects,  some  formality  was  required  to  provide  consistency. 
Documentation  was  prepared  to  describe  the  role  playing  event,  basically 
who  did  what  and  when.  It  evolved  into  a  time  based  description  of  the 
roles  of  the  crew  members  in  accomplishing  a  mission.  Exposing  several 
crews  to  the  same  scenario  provided  an  opportunity  to  obtain  reinforcement 
of  some  crew  reactions,  a  useful  output.  The  psychologists  attached  to 
our  teams  really  capitalized  upon  the  opportunity  to  acquire  subjective  data 
which  assessed  the  utility  of  a  set  of  displays.  Of  particular  merit  was 
the  fact  that  the  subjects  were  viewing  the  displays  in  the  context  of  their 
intended  use.  Over  the  years  this  progressed  to  the  point  of  developing  a 
narrative  covering  the  entire  menu  of  missions  to  be  performed.  Subse¬ 
quently,  we  put  all  of  the  missions  of  a  planned  vehicle  in  one  flight  plan. 
Although  unrealistic  in  terms  of  operations  (no  one  flight  could  be  ex¬ 
pected  to  perform  every  one  of  the  missions  for  which  the  system  had 
capability) ,  it  was  a  way  of  compressing  all  the  design  challenges  into 
one  bundle  and  providing  a  consistent  means  of  addressing  these  problems. 
This  became  known  as  our  "DESIGN  SCENARIO." 

Dynamic  Review. — We  assembled  all  of  the  proposed  device  designs 
into  a  cockpit  mockup  and  flew  the  mission  as  defined  by  the  Design 
Scenario.  .  .  (a  paper  mockup,  imaginary  flight,  real  crews).  As  glitches 
were  discovered,  we  could  redraw /relocate /redesign  and  refly.  We  then 
were  in  a  position  to  specify  details  which  would  otherwise  be  uncontrolled. 
As  teams  developed  changes  we  could  readily  assess  the  impact  upon  crew 
and  mission. 


We  then  were  in  a  position  to  specify  details  which  would  other¬ 
wise  be  uncontrolled.  As  changes  were  proposed  we  could  readily  consider 
the  impact  upon  crew  and  mission. 

Although  made  only  of  paper,  several  different  kinds  of  problems 
can  be  effectively  addressed  in  this  mockup.  (1)  Our  description  of  the 
mission  (the  real  crews  will  catch  glitches);  (2)  Crew  size  and  crew  role 
(several  different  design  concepts  can  be  tried  and  an  informed  judgment 
made  as  to  their  probability  of  success.)  This  is  a  way  for  capitalizing 
upon  the  thousands  of  experiences  of  the  operational  crews.  These  experi¬ 
ences  will  be  implicit  in  the  comments  and  judgments  of  the  subject  crews. 
(Thousands  of  experiences,  the  details  of  which  are  not  remembered,  provide 
for  the  intuitive  judgment  of  experts  upon  which  we  all  rely  so  heavily.) 

The  weight  and  value  of  the  collective  experiences  of  many  crews  can  be 
brought  to  bear,  constructively,  in  the  design  process,  even  before  we 
start  bending  metal;  (3)  A  base  line  for  the  selected  approach  against 
which  all  future  variations /compromises  can  be  examined  for  effects  upon 
mission  performance  and  quantified;  (4)  A  base  for  developing  and  ex¬ 
tending  this  approach  for  assessing  system  and  mission  effectiveness  into 
more  sophisticated  and  sensitive  areas  wherein  all  relevant  dynamics  and 
functions  can  be  examined  (simulation  and  flight  test);  (5)  Geometry  (crew 
position,  windows,  panels,  ingress  and  egress  can  be  examined  in  context.) 

Component  Development  and  Test  (Research,  Development) .—In  the 
design  of  instruments,  display  content  and  format  are  crucial  factors  in  the 
ultimate  performance  as  is  the  accuracy,  response,  repeatability,  etc.  of 
the  mechanism.  Bench  testing  of  the  mechanism,  in  a  systems  context  is 
widely  used  (but  really  in  a  limited  systems  context) .  The  equivalent  for 
display  content  and  format  has  not  reached  the  same  level  of  maturity.  Part 
task  simulation,  naive  subjects,  even  experienced  subjects,  left  us  with  an 
unsatisfied  feeling.  General  purpose  digital  computers  and  our  crew  station 
mockup  provided  us  With  the  means.  The  Design  Scenario  and  our  mock 
flights  provided  the  scheme.  Using  laboratory  equipment  and  general  pur¬ 
pose  CRT's,  a  proposed  instrument  could  be  represented  in  proper  context 
in  the  cockpit  and  the  experiments  could  be  performed  in  the  context  of 
the  mission  to  be  performed.  What  we  were  able  to  do  is  not  part  task  but 
full  task,  part  mission.  The  segment  of  mission  selected  permitted  exer¬ 
cising  the  display  to  the  full  extent  necessary,  with  the  crew  and  in  the 
context  of  associated  work  and  distractions  (or  at  least  with  a  higher  degree 
of  fidelity  than  otherwise  used).  The  cost  for  such  a  setup  is,  of  course, 
substantially  higher  than  part  task  simulation.  The  difference  in  quality 
does,  in  our  opinion,  justify  the  time  and  cost.  However,  once  in  place  the 
cost  and  time  per  subsequent  experiment  becomes  highly  cost  effective  and 
time  efficient  compared  to  conventional  laboratory  approaches  while  retaining 
the  much  higher  level  of  value  (Ref.  8). 

Thus  we  have  established  a  design  method  which  can  continually 
expose  the  effect  of  subsystem  change  and  compromise  upon  the  total  system 
performance. 


An  extremely  important  factor  is  that  system /mission  compromises 
are  taking  place,  reflecting  adjustments  to  the  myriad  of  subsystem  changes. 
Now  with  this  scheme  of  a  design  mission  the  changes  can  be  readily  re¬ 
flected  down  to  the  component  design  level  in  a  manner  which  permits  the 
the  designer  to  quantify  the  impact  upon  his  own  design.  While  control 
from  a  system  level  upon  component  details  is  implied ,  it  is  really  control 
of  the  detail  structuring  of  the  problem  and  criteria,  leaving  considerable 
freedom  for  the  individual  engineers  and  scientists  in  exercising  their  own 
judgment  for  producing  the  most  suitable  product. 

Crew  System  Design. — All  design  can  be  improved  by  an  iterative 
approach  (each  mistake  discovered  can  be  corrected  in  the  next  attempt). 

In  the  crew  system  case,  it  is  hoped  that  each  iteration  will  proceed  to  a 
greater  level  of  detail  and  involve  fewer  corrections  to  the  thread  of  the 
prior  design.  As  the  design  of  components  and  subsystems  progresses, 
the  scheme  of  assessing  mission  effectiveness  and  system  suitability  must 
also  progress  to  deal  with  the  increasing  depth  of  detail  and  breadth  of 
concern.  The  individuals  most  able  to  do  this  are  those  steeped  in  all 
aspects  of  the  mission  performance,  operational  employment,  system  func¬ 
tioning.  Of  course  this  has  been  the  thrust  of  training  and  responsibility 
of  the  core  group.  They  must  insure  that  mission  scenario  details  are 
expanded  upon  and  validated. 

Details  of  system  functioning  must  progress  from  abstract  con¬ 
ception  to  the  hard  realities  of  what  the  equipment  and  crew  can  do  and 
what  the  real  world  exposure  is  going  to  bring  in  terms  of  geographic 
environment,  weather  environment,  threat  environment,  force  structure, 
operational  constraints  and  economic  and  political  factors  and  constraints. 

The  narrative  mission  description  evolves  in  great  detail  and  many 
forms.  The  process  of  advancing  this  can  be  extremely  confusing  and 
challenging.  It  poses  considerable  demands  upon  the  core  group  in  exer¬ 
cising  tact  in  human  relations  and  challenges  the  technical  breadth  and 
capacity  of  the  core  group.  The  harmonization  of  the  efforts  of  thousands 
of  diverse  personalities  in  hundreds  of  locations  without  face  to  face  con¬ 
tact  is  one  of  the  most  imperative  ingredients  of  a  successful  weapon  system 
design.  We  are  not  assuming  any  authority  for  the  weapon  system  design, 
we  are  concerned  with  a  subsystem  that,  in  effect,  has  tentacles  into  or 
is  affected  by  every  aspect,  every  facet,  of  the  weapon  system  design.  We 
must  be  aware.  We  must  be  responsive  to  all  elements  of  the  weapon 
system  if  the  crew  is  to  be  able  to  demand  and  get  every  bit  of  performance 
possible  from  the  weapon  system. 


PROVE 


Our  ability  to  learn  something  useful  about  the  potential  value  of  a 
crew  station  design  improved  with  the  flying  of  the  paper  rnockup  (a 
dynamic  review) .  To  enhance  the  learning  comparison  of  subsequent 
iterations,  it  was  necessary  to  establish  a  formalized  structure  for  this 
review  and  test.  Although  almost  totally  subjective,  a  degree  of  quanti¬ 
fication  is  possible  through  the  use  of  rating  scales.  The  Cooper-Harper 
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rating  scale  for  handling  qualities  (Ref.  9)  is  widely  used  and  accepted. 

A  similar  scheme  for  our  interests  seemed  to  work  quite  well.  As  you 
would  expect,  a  questionnaire  suitable  for  standardization  doesn't  get 
established  on  one  try.  We  had  to  determine  what  were  the  meaningful 
and  useful  questions  to  ask.  We  could  and  did  (still  do)  use  tests  of 
readibility,  interpretability ,  etc.,  in  the  individual  design  efforts.  What 
we  needed  for  the  gestalt  of  the  crew  system  was  a  test  (or  battery  cf 
tests)  which  would  be  sensitive  to  system  performance  factors.  They 
should  identify  and  measure  factors  which  designers  can  respond  to  in 
pursuing  improvement. 

There  are  several  kinds  of  information  which  could  be  useful. 

(For  example,  examining  each  mission  task  -  could  it  be  performed  at  all  - 
did  it  strain  the  capability  of  the  crew?  If  difficulties  were  experienced, 
what  was  it  that  contributed  most  to  the  difficulties?)  The  mock  mission 
flight  by  experienced  crews  provided  to  them  the  necessary  stimulus  and 
reminders  to  permit  very  substantial  contributions  along  this  line  -  usually 
in  a  narrative  format  -  often  in  debriefing  interviews  (or  bull  sessions) . 

The  crew  judgment  was  (and  is)  based  upon  their  unique  experience. 
We  have  a  comparable  situation  with  respect  to  experimenters.  As  they 
accumulate  experience  in  observing  crews,  they  also  develop  an  "intuitive" 
ability  which  differs  from  that  of  the  crew  in  that  the  intuition  is  derived 
from  observing  many  subjects  as  contrasted  to  the  single  subject  experience 
of  a  crew  member  (at  least  it  is  self  observation,  slightly  modified  by  obser¬ 
vation  of  crew  mates) . 

The  use  of  experimenters'  subjective  judgment  is  a  relatively  new 
idea  and  the  technique  seems  to  have  positive  advantages.  The  initial 
questionnaires  were  useful  but  are  still  evolving.  (Should  we  call  the 
experimenters  second  order  subjects?) 

The  data  acquired  from  the  crews  and  experimenters  in  the  role 
playing  exercises  with  the  paper  mockup  gave  us  a  tremendous  lift  and 
sense  of  progress.  We  felt  as  though  we  were  developing  more  insight  into 
what  makes  a  good  operation.  A  very  personal  feeling  of  satisfaction  was 
evoked  by  the  progress  in  improving  the  cockpit  and  the  crew  performance 
without  addressing  the  flying  task  or  simulation  of  the  flight  control  system. 
This  serves  to  dramatically  emphasize  the  point  that  over  50%  of  the  work¬ 
load  and  challenges  upon  the  crew  are  tasks  other  than  flying.  The  flying 
task  is  so  important  that  concern  with  it  has  tended  to  obscure  the  growth 
in  significance  and  workload  of  the  other  crew  responsibilities.  In  the 
process  being  discussed,  control  system  teams  have  been  pursuing  a  parallel 
track  to  that  of  the  crew  system  team  -  with  lots  of  communication. 

With  the  conclusion  of  the  formal  dynamics  review  (paper  mockup 
phase),  the  design  team  had  a  conceptual  design  in  which  they  could  have 
a  high  degree  of  confidence.  They  were  in  a  position  to  write,  in  greater 
detail,  of  the  characteristics  needed  and  constraints  required  for  each  of 
the  cockpit  devices  and  the  functioning  of  the  supportive  system.  If  the 
communication  link  has  been  strong  and  effective,  nothing  will  come  as  a 
surprise  to  all  of  the  affected  design  groups  (comm,  nav,  fire  control. 


flight  control,  instruments,  etc.)  It  is  to  be  hoped  that  they  have  had 
a  consultive  participation  all  along.  (It's  in  their  own  interest  to  con¬ 
tribute  and  to  get  this  information.) 

The  use  of  a  paper  mockup  does  not  provide  for  the  kind  of 
numbers  so  dear  to  the  heart  of  an  engineer.  It  does  provide  numbers 
(subjects,  subjective  quantification,  time  durations,  errors,  etc.)  for  the 
psychologist.  This  is  interesting  (and  appropriate).  The  emphasis  is  upon 
functioning,  or  functional  relations  -  a  description  of  what  ought  to  be. 

It  provides  the  boundaries,  the  scoping  necessary  for  the  engineers  to  ply 
their  trade.  It  is  far  different  from  the  conventional  use  of  behavioral 
scientists  (which  is  frequently  for  evaluation  after  the  fact  of  design  or 
to  produce  generalizable  handbook  type  data.)  It's  a  role  not  clearly 
recognized  and  defined  -  yet  it  may  be  the  most  significant  one. 

The  completion  of  the  dynamic  review  of  the  mockup  is  as  close  as 
we  came  to  a  formal  completion  of  the  problem  definition  stage  (preliminary 
investigation).  At  this  point,  we  have  a  fair  degree  of  formalization  in 
architecture  and  in  functioning  and  in  partition.  The  customer  has  a  good 
appreciation  of  the  planned  functioning  and  capabilities  of  the  weapon  system. 
The  various  subsystem  teams  are  quite  well  oriented  so  as  to  pursue  their 
individual  design  concepts  to  arrive  at  the  equations,  the  gains,  the  values, 
the  specificity  of  how  the  hardware  will  be  built  and  what  it  will  do. 
Breadboard  equipment  is  a  possibility  but  commitment  to  final  fabrication 
is  not.  The  next  level  of  sophistication  in  assessment  is  required.  Basi¬ 
cally,  it  is  a  look  at  how  well  all  the  equations  and  numbers  are  going  to 
relate  to  the  functional  concepts  of  the  paper  mockup  stage.  In  a  word  - 
simulation.  If  mockup  was  "formative  stage"  then  simulation  is  "definitive 
stage. " 

Simulation. — The  Simulation  is  not  just  a  system  assessment  effort. 

It  is  a  family  of  efforts.  The  core  group  interest  is  in  analyzing  the  system, 
while  the  interests  of  the  subsystem  teams  are  in  analyzing  their  own  special 
areas.  It's  a  period  of  iterative  testing.  First,  the  overall,  fly  the  mission 
and  identify  weak  points  or  needed  subsystem  adjustments.  Second,  en¬ 
gineering  employment  for  each  of  the  subsystems  (man-machine,  r.av, 
flight,  armament,  etc.)  Third,  iterate  steps  one  and  two  until  a  satis¬ 
fied  concensus  is  reached. 

Some  problems  may  be  taken  from  the  system  simulator  for  intensive 
experimentation  and  investigation  on  smaller,  specialized  simulation  equipment. 

If  this  sounds  expensive  -  believe  it.  But  you  have  only  to  do  it 
once  to  become  convinced  that  it  is  worth  it.  AND  that  it  is  more  cost 
effective  than  the  past  conventions.  The  most  startling  thing  is  that  a 
large  cost,  heretofore  hidden  (by  being  included  in  larger  items),  has 
become  identifiable.  Once  identified,  it  becomes  subject  to  efforts  to  make 
it  cost  effective.  The  dollar  value  sounds  large  but  we  made  an  effort  to 
compare  it  against  the  costs  of  the  conventional  approach.  Considering 
the  cost  of  corrections  normally  necessary  after  building,  we  estimated, 
conservatively,  that  there  would  be  a  true  improvement  of  50%  in  the  cost 
picture.  Unfortunately,  management  does  not  normally  have  prior  costs 
broken  out  to  permit  direct  comparisons. 
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It  has  been  out  of  learning  programs  such  as  this  that  concepts 
for  system  growth  and  new  system  concepts  have  evolved  (e.g.,  fire- 
flight  control  systems,  flight-propulsion  control  systems,  crew  adaptive 
systems) . 

In  the  simulation  work  we  strove  for  fidelity  and  realism.  Many 
little  things  that  don't  impinge  upon  your  consciousness  contribute  to 
feelings  of  realism.  Each  program  which  we  conducted  improved  in  this 
respect  and  in  each  one  we  resolved  to  improve  the  next.  Our  preflight 
planning  and  briefing  conformed  to  the  real  operational  performance. 

Flight  clothes  were  used.  Cockpit  checkout  and  preparations  were  as 
realistic  as  possible.  Background  noise  was  simulated.  Radio  traffic, 
static,  etc.  ,  was  as  realistic  as  possible.  All  possible  communication 
points  were  "manned"  by  experimenters.  Communication,  for  example, 
simulated  the  effects  of  a  helicopter  upon  a  speaker  (he  thumped  his  chest 
while  talking  to  give  the  effect).  Whether  it  was  the  validity  of  the  situ¬ 
ation  or  the  observed  intent  and  seriousness  of  the  experimenters  to  be 
thorough  (a'la  the  Hartford  experiment),  we  don't  know,  but  the  oper¬ 
ational  crews  were  very  responsive  and  it  worked.  Whatever  the  reason, 
the  continued  increases  in  realism  produced  an  improved  situation.  Our 
interest  in  realism  extended  to  all  stations  and  crew  members.  For  a 
loadmaster  we  not  only  mocked  up  his  station  but  an  entire  cargo  with 
lashings  and  tie  downs. 


Figure  3.  Cargo  Compartments 
(Real  and  Simulated) 


The  multiple  use  of  the  simulator  includes  use  of  it  for  re¬ 
examining  crew  roles  and  duties  and  re-analysis  of  the  scenario  based 
upon  measured  values. 

We  had  started  with  a  narrative  of  a  scenario  that  included  all 
mission  tasks  (the  Design  Scenario) .  Using  the  observations  made  in 
similar  type  flights  and  with  the  participation  of  crews  experienced  in  the 
mission,  a  time  based  description  had  been  devised  which  included  details 
of  functions  to  be  performed  (by  machine  or  by  man).  As  the  new 
system  evolved  we  adjusted  this  to  be  compatible  with  the  descriued 
functioning  of  the  new  system.  In  the  initial  layouts  of  .the  paper  mockup 
we  sometimes  started  from  a  preferred  crew  size  and  crew  role  statement. 

In  some  instances  crew  size  and  role  was  not  constrained  and  then  we 
tried  "ranging  shots"  of  high,  intermediate  and  low  crew  sizes  so  as  to 
span  the  probable  crew  size.  The  detailed  scenarios  could  be  adjusted 
to  assume  roles  and  duties  for  each  of  the  crew  sizes.  However,  these 
adjustments  were  estimates  only.  We  "flew"  the  estimates  to  check  them 
out  -  corrected /adjusted  and  reflew.  For  this  part  of  the  design  effort 
we  needed  operational  people  who  participated  on  a  contractual  basis.  A 
considerable  amount  of  experienced  and  qualified  judgment  was  needed  to 
examine  and  adjust  the  crew  duties  and  responsibilities.  The  task  re¬ 
quired  an  inordinate  amount  of  details  and  alternatives  to  be  juggled. 

Many  of  the  very  critical  considerations  for  the  process  are  not  to  be 
found  documented. 

Trial  after  trial  in  the  mockup  led  to  detail,  time  based  descrip¬ 
tions  of  the  activities  of  each  crew  member,  the  functioning  of  the  equip¬ 
ment  and  the  relation  to  the  mission  for  each  design.  Each  design  included 
crew  size  and  equipment.  The  trade  offs  included  changes  in  crew  roles; 
crew  complement;  crew  size;  equipment;  automaticity ;  mission  tactics. 

The  number  of  variables  and  the  range  of  effects  resulting  from 
changes  in  just  one  variable  became  so  great  that  formalized  procedures 
never  got  defined. 

We  had  to  rely  upon  the  collective  judgment  of  the  pilots,  engineers, 
psychologists  of  the  core  group  to  bring  the  scope  of  possibilities  down  to 
a  manageable  set  of  alternatives. 

With  the  determination  of  specie  alternative  designs  (crew  size 
and  role  is  a  design  variable),  the  scenarios  could  be  "firmed  up." 

While  the  narrative  design  scenario  is  always  applicable,  the  details  in 
the  design  scenario  and  the  bench  mark  include  the  unique  aspects  of  each 
design  alternative. 

(During  actual  practice,  an  observer  might  be 
confused  and  less  than  confident.  The  trial  and  error 
design  methods  of  many  areas  might  not  instill  confidence 
if  observed.  In  our  case,  the  activity  is  much  more  open 
and  observable  since  a  mockup  is  used.  Additionally,  many 
more  people  feel  qualified  to  speak  about  the  cockpit  -  it 
seems  like  every  person  in  the  aviation  community  feels 


qualified  as  an  expert  on  cockpit  matters.  This  is  not  a 
complaint,  merely  an  observation  on  the  degree  of  atten¬ 
tion  that  cockpit  work  engenders.) 

This  activity  was  a  lead-in  to  the  cockpit  layout.  Of  course  we 
started  with  some  hypothetical  layouts,  but  they  were  trials  only,  something 
of  substance  to  allow  work  to  proceed.  The  determination  of  specific  crew 
and  equipment  parameters  provides  a  base  against  which  a  specific  cock¬ 
pit  design  is  evolved.  This  phase  draws  on  additional  sets  of  experts  and 
a  whole  host  of  additional  tradeoffs.  We  are  allocating  space  and  position 
to  switches,  knobs,  displays,  lights,  annunciators  -  literally  hundreds  of 
elements.  Sometimes  it  seems  each  element  has  its  own  champion.  We  have 
had  meetings  with  all  of  these  present.  Each  person  to  explain  the  func¬ 
tioning  of  the  device  to  the  others  and  all  to  hear  the  rationale  for  the 
design  patterns  established  and  the  framework  (mission /design  scenario) 
for  employment.  A  meeting  like  that  is  an  impressive  experience  in  human 
dynamics. 

Large  group  discussions  can  stimulate  thoughts  and  bring  out  points 
that  might  otherwise  be  missed.  So  it  was  useful  to  have  them  once  in  a 
while.  In  our  early  efforts  they  were  a  real  adjunct  to  our  learning  pro¬ 
cess.  As  we  learned  from  them  we  could  incorporate  the  results  into  our 
process  and  therefore  be  less  dependent  upon  those  kinds  of  meetings  in 
subsequent  programs.  (Don't  exclude  them  completely  -  but  one  may  be 
all  you  can  stand.)  In  general,  the  impact  of  this  experience  was  to  be 
more  explicit  in  our  scenarios  and  in  detailing  the  functioning  of  equipment. 

This  step  developed  two  themes  for  time  based  description  of  the 
mission:  (1)  The  crew  oriented  viewpoint  -  who  does  what,  when,  and 

with  what,  and  (2)  the  equipment  oriented  -  what  is  each  gadget  doing  - 
what's  happening  in  the  data  flow  (it  is  a  data  system  management  orien¬ 
tation).  The  development  of  the  first  is  crew  dominant  while  the  second  is 
engineer  dominant. 

This  step  has  gone  a  long  way  toward  establishing  guidelines  and 
constraints  for  the  component  people.  That's  not.  to  imply  that  they  have 
been  idle,  waiting  for  this.  Their  design  work  has  been  progressing  in 
parallel  with  this  activity.  They  have  ascertained  many  design  options  and 
this  present  activity  of  the  core  group  provides  a  firm  basis  for  selection 
among  those  design  options  and  guidance  for  further  refinement  of  the 
options  selected. 

Details  of  the  display  had  been  hypothesized.  Now  a  clearly  struc¬ 
tured  situation  can  be  presented.  That  is,  the  specifics  of  a  mission  task, 
the  information  required  and  the  detail  decision  needed.  Maximizing  the 
man-machine  relationship  can  be  pursued  effectively  within  this  framework. 
(Past  activities  frequently  looked  to  maximize  man-machine  irrespective  of 
mission  effects  -  or  with  the  attitude  that  "What's  good  for  man-machines 
is  good  for  the  system.") 


Part  task  simulation  of  full  mission  activity  can  be  focused  upon 
quite  explicit  goals.  Tailoring  of  generalized  human  factors  principles 
to  the  specifics  of  this  situation  is  now  a  practical  step. 

This  just  about  reaches  the  limits  of  utility  of  the  paper  mockup. 

The  level  being  approached  in  the  system  design  requires  more  sensitive 
system  tests  and  more  fidelity  in  representing  the  system  that  is  to  be 
tested. 

This  can  be  an  evolutionary  type  of  improvement.  For  example, 
in  one  program  we  proposed  a  new  combination  of  nav  and  comm  control. 

It  was  an  integrated  control  head.  We  breadboarded  it  and  put  the  func¬ 
tioning  control  head  and  display  units  into  the  paper  mockup  for  "first 
look"  in  the  system /mission  context.  Several  were  tried  and  the  experience 
stimulated  some  creative  and  useful  variations. 

Where  more  complex  situations  existed,  full  simulation  was  used. 

For  example,  a  subsystem  involving  multipurpose  electronic  display  and 
multifunction  switching  for  management  of  weapons  stores.  Adequate  in¬ 
vestigation  required  that  the  full  dynamics  of  a  mission  situation  be  represented. 
A  mission  scenario,  as  I've  described  it,  provided  the  framework  against 
which  to  investigate  the  problem.  A  fully  equipped  cab  and  simulator  pro¬ 
vided  the  equipment.  Operational  crews  flew  applicable  mission  segments 
wherein  the  total  mission  situation  and  loading  was  presented.  This  was  a 
full  task,  part  mission  experiment.  The  center  of  focus  was  on  the  design 
of  a  specific  subsystem,  but  the  framework /structure  used  provided  for  the 
total  system  context. 

The  results  had  face  validity  for  the  casual  observer  and  were 
highly  defensible  for  the  more  perceptive  reviewers. 

As  our  paper  mockup  phased  out,  the  activity  on  crew  station 
component  design  moved  toward  the  climax  of  detailed  design  evaluation 
(at  the  subsystem  level  but  in  a  systems  related  manner). 

Naturally  enough,  a  successful  and  satisfying  experience  in  one 
program  led  to  repetition  and  expansion  in  subsequent  programs.  Since 
our  organization  was  R&D  we  were  not  committed  to  production  schedules 
and  we  were  resource  limited.  For  each  program  there  was  a  prioritization 
of  effort  on  the  basis  of  advancing  understanding  and  value  of  results. 
Consequently,  many  of  the  examples,  which  we  have,  tend  to  be  from 
isolated  piecemeal  programs. 

Rising  personnel  costs  raise  questions  about  reducing  crew  size. 

Greater  and  more  diverse  vehicle  and  weapon  capabilities  raise  questions 
about  the  crew  ability  to  cope  and  employ.  Rising  equipment  costs  and 
increasing  complexity  of  equipment  raise  questions  about  the  inclusion  of 
such  equipment.  In  all  cases  the  tradeoff  process  must  look  at  the  effects 
of  any  decision  upon  mission  effectiveness.  It  should  provide  this  oppor¬ 
tunity  prior  to  vast  expenditures  for  prototype  equipment. 


Considerations  such  as  these  have  led  to  the  need  for  a  more 
extensive  and  more  rapid  expansion  of  (our  activities)  this  process. 

In  recent  years  we've  been  able  to  put  many  of  the  pieces  to¬ 
gether  into  more  complete  programs  (Ref.  10,  11). 

It  takes  a  large  program  to  support  the  subsequent  steps  in  this 
methodology. 

We  have  followed  a  theme  of  development,  refinement  and  in¬ 
creasing  detail  in  describing  the  design  mission  (the  design  scenario,  to 
be  used  as  the  bench  mark  in  assessing  the  effectiveness  of  proposed 
systems).  While  the  total  system  must  be  represented,  the  method  is 
sensitive  and  definitive  to  variables  in  the  crew  system.  Representation 
of  a  fully  functioning  system  and  the  ability  to  fly  it  against  a  realistic 
mission  problem  requires  large  scale  simulation  facilities  (or  access  thereto) 
for  substantial  time  periods. 

So  far  as  procedure  is  concerned,  simulation  uses  the  same  basic 
approach  as  the  paper  mockup  effort,  only  it  is  more  complicated  and 
demanding.  For  our  first  experience,  one  of  the  traumatic  adjustments 
came  about  because  we  had  equipment  and  experienced  simulation  people. 

The  people  and  equipment  had  been  oriented  to  flight  control  problems. 

The  natural  thought  was  to  expand  that  to  include  some  of  the  subsystems. 
However,  the  adjustments  were  far  more  significant  than  mere  additions. 
Gross  approximations ,  so  perfectly  satisfactory  before  were  not  recognized 
initially  as  inappropriate.  Test  sensitivity  requirements  were  drastically 
different,  as  were  test  conditions.  The  part  task  aspects  of  flight  control 
simulation  are  several  orders  of  magnitude  more  complex  than  what  is  con¬ 
ventionally  visualized  as  part  task  -  so  much  so  that  many  took  to  thinking 
of  it  as  though  it  were  the  full  thing.  In  a  nutshell,  the  transition  asked 
was  from  part  task,  part  mission  simulation  to  full  task,  full  mission 
simulation. 

When  discussing  the  mockup,  I  referred  to  it  as  the  "paper"  mockup 
to  emphasize  that  it  could  be  cheap  and  simple.  Over  the  years  we  have 
actually  used  many  variations  from  paper  (literally)  to  rather  substantial 
metal  structures.  "Foam  core"  seems  to  be  a  useful  medium  for  the  initial 
trials. 
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Figure  4.  Mockups 


We  have  substantial  shells  for  different  classes  and  sizes  of 
vehicles  in  which  we  can  readily  install  foam  core  panels  and  pictures. 

As  time  passes  and  designs  are  made  more  firm,  the  foam  core  and  paste 
ups  can  be  replaced  with  more  durable  structures  and  devices.  The 
mockup  can,  therefore,  grow.  It  is  possible  for  it  to  be  used  as  the 
simulator  cab  if  the  shell  and  basic  structure  are  suitable. 

As  time  passed  we  felt  the  need  to  identify  the  level  of  sophis¬ 
tication  of  the  cabs  used.  We  established  three  levels  (Class  A,  B,  C) 
(Appendix  A).  The  simplest  could  suffice  for  the  initial  work,  but  in  the 
final  stages  of  design  and  of  evaluation  many  features  had  to  be  precisely 
simulated  and  controlled. 


SECTION  II 


OVERVIEW 


Chapter  I  reviewed  some  of  the  history,  rationale  and  experiences 
leading  to  our  present  views  of  an  appropriate  methodology  for  crew 
system  design,  test  and  evaluation.  It  is  the  intent  from  this  point  to 
discuss  the  pattern  of  activities,  the  dependent  relationships,  the  time 
relationships  and  significant  ramifications  and  relations  beyond  the  crew 
system  domain. 

It's  been  said  before,  but  can't  be  repeated  too  often.  The  work 
does  not  fit  into  the  neat  independent  blocks.  We  use  block  diagrams  as 
an  inadequate  tool  for  communication.  It  helps  to  focus  on  a  point  we 
wish  to  draw  to  your  attention.  We  use  lines  to  connect  blocks  for  pur¬ 
poses  of  showing  dependent  relationships  and  the  logic  of  process.  In 
reality  the  interflow  exists  universally.  A  real  diagram  would  probably 
be  like  looking  at  a  huge  platter  of  spaghetti  and  meat  sauce.  (For  those 
of  you  who  understand  this  point,  please  excuse  the  exhortation.  Due  to 
the  difficulties  of  communication,  we  have  found  it  necessary  to  belabor 
this  again  and  again.) 

The  process  follows  that  very  basic  idea  of  Define  the  Problem, 
Develop  a  Solution  and  Prove  the  Solution.  We  have  found  it  convenient 
to  use  this  division  in  discussions  because  it  conforms  to  significant 
changes  that  actually  take  place.  The  nature  of  the  work  changes,  the 
skill  requirements  change,  the  magnitudes  of  cost  and  manhours  involved 
change  in  accordance  with  these  divisions.  Even  the  time  factors  change. 

It  is  useful  to  note  that  starting  a  program  such  as  this  is  not  a 
commitment  to  the  total  program.  Each  of  the  three  basic  phases  can  estab¬ 
lish  plateaus  where  program  termination  will  not  be  harmful  in  the  sense  of 
wasted  expense  and  effort.  Each  phase  provides  a  milestone  worthy  of 
achievement. 

Another  item  to  be  noted  is  the  role  of  test  and  evaluation.  A 
structure  for  testing  is  an  implicit  part  of  the  methodology  and  the  test/ 
evaluation  process.  It  starts  on  DAY  ONE  along  with  the  technical  effort. 

It  is  not  just  an  after  the  fact  process. 

Testing  requires  a  structured  plan  for  the  acquisition  of  data 
describing  the  characteristics  and  behavior  of  a  component,  a  subsystem 
or  a  system.  Likewise,  evaluation  requires  a  structured  plan  for  the 
determination  of  the  criteria  against  which  the  test  data  will  be  assessed. 
Each  phase  of  this  program  requires  validation  (test,  evaluation  and  ap¬ 
proval),  starting  with  the  initial  statement  of  the  problem.  As  a  program 
progresses  tests  will  accumulate  data  on  the  components,  the  subsystem  and 
on  the  total  system.  All  of  the  tests  provided  data  for  the  final  decision 
process.  There  is  a  pattern  to  effective  testing  so  that  the  accumulation 
can  contribute  effectively  to  the  final  validation  process. 
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DEFINE  THE  PROBLEM 


The  starting  point  is  identified  in  our  diagram  as  the  "SON" 
(Statement  of  Need).  It  is  generally  understood  that  such  a  formal  state¬ 
ment  should  provide  as  much  leeway  as  possible  for  the  vehicle  designers 
who  will  compete.  The  objective  is  to  provide  the  greatest  opportunity 
for  creativity  and  innovation  in  advancing  our  capabilities.  There  IS  a 
need,  an  anticipated  or  actual  deficiency  in  our  force  capabilities.  It  is 
possible  that  in  some  plant  or  organization  an  idea  exists  that  would 
merit  a  totally  different  approach  to  dealing  with  the  problem.  However, 
lacking  this,  the  originators  do  have  in  mind  a  way  to  deal  with  the 
need.  The  SON  is  a  starting  point  for  exploring  ideas,  not  necessarily 
the  point  for  starting  development.  In  the  crew  systems  arena  we  are 
not  going  to  advance  capability  until  we  know  the  challenges.  This  is 
perhaps  where  the  real  distinction  between  it  and  other  subsystems  exist. 
Radio,  navigation,  radar,  etc.,  are  all  physical  areas.  The  challenge  to 
expanding  their  technology  is  definable  in  the  terms  of  physical  science, 
generally,  irrespective  of  the  application.  In  the  life  science  area  the 
challenge  is  to  know  and  define  the  behavioral,  physiological  and  psycho¬ 
logical  capabilities  and  limitations  of  a  biological  element. 

The  crew  systems  challenge  is  how  to  use  all  of  this  knowledge 
for  the  successful  employment  of  warfare  systems.  The  challenge  is  more 
in  the  intellectual,  cognitive  functioning  of  a  complex  of  men  and  machine 
doing  a  job.  A  job  where  gross  performance  characteristics  are  known 
but  detail  perturbations  can  not  always  be  predicted.  The  intangible 
feeling  is  what  gets  you.  You  have  to  ask  about  a  lot  of  things  before 
you  can  know  that  some  particular  piece  of  info  really  isn't  needed. 

The  SON  is  the  trigger  to  cause  people  to  look  for  improved  air¬ 
frame  and  propulsion,  improved  ordnance,  etc.,  etc.  It  should  also  be 
thp  trigger  to  look  for  improvements  in  the  total  mission  performances 
through  proper  matching  and  use  of  crew  and  equipment. 

The  problem  must  be  explored  from  the  perspective  of  the  crew 
and  the  intentions  of  the  warfare  planners. 

There  are  several  boxes  used,  all  to  the  effect  of  understanding 
the  problem  in  breadth  and  depth  so  as  to  BE  in  a  position  to  be  creative 
and  responsive  in  crew  system  design.  The  milieu  for  airframe  design  is 
the  atmosphere,  the  milieu  for  radio  and  radar  is  the  electromagnetic 
environment,  the  milieu  for  crew  systems  is  the  mission.  It  is  not  a 
basic  physical  condition  of  nature  subject  only  to  the  laws  of  nature. 

It  is  an  artifact  whose  characteristics  are  subject  to  the  whims  of  man 
(of  course  within  the  laws  of  nature). 

Because  of  this  difference,  exploring  the  problem  is  a  search  for 
things  (details,  data)  that  may  be  of  value.  Because  of  its  nature  it  is 
not  necessarily  responsive  to  the  same  questions  and  measurements  each 
time  around.  Efficiency  of  interview  needs  a  structure  but  we  can't  be 
assured  the  structure  is  appropriate  for  the  next  interview. 
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Face  to  face  interviews  at  all  levels  of  the  customer  organization, 
from  the  crew  to  the  command  level  is  desirable.  Participation  in  the 
current  equivalent  of  the  desired  operations  is  valuable.  All  with  the 
objective  of  getting  a  gut  feel  for  all  desires  of  the  customer  and  all 
ramifications  of  the  problem. 

Understanding  the  problem  in  infinite  detail  is  40%  of  the  challenge 
in  building  a  system.  Knowing  what  constitutes  the  real  scope  of  the 
problem  is  50%. 

We  have  said  a  test  is  a  process  for  getting  data.  The  test  tech¬ 
nique  in  this  instance  is  the  interview.  The  data  must  be  organized  and 
validated  ("Is  this  what  you  want?")  to  insure  that  subsequent  efforts  are 
properly  pointed. 

SOLVE 

There  is  a  finite  indication  on  the  chart  for  transition  from  Define 
to  Solve.  It's  really  a  change  in  tempo  and  emphasis.  Tentative  designs 
are  useful  in  exploring  the  problem.  They  help  uncover  miscues  in  com¬ 
munication.  They  may  reveal  to  the  initiator  that  the  original  request  had 
as  implied  needs,  features  not  really  necessary.  The  design  may  surface 
discussion  of  needs  or  constraints  not  previously  mentioned. 

So  far  as  the  Solve  orientation  goes,  the  checking  of  the  trial  designs 
helps  to  steer  the  design  themes  into  acceptable  channels. 

Initial  design  schemes  are  merely  grossly  simplified  discussions  of 
systems  concepts.  As  the  requestor  and  the  responder  progress  into  a 
satisfactory  theme  the  approach  can  be  partitioned  and  teams  initiated  for 
these  partitions.  In  each  team  an  action  takes  place  equivalent  to  that 
which  the  requestor  responder  went  through.  Trial  designs  for  each  of 
the  teams  checked  back  against  the  systems  concept  of  the  responder  (and 
core  group). 

Thus  the  activity  grows  and  branches  but  always  retains  a  close 
check  to  the  overall  theme  of  the  design.  This  branching  growth  draws  in 
more  people  and  proceeds  to  levels  of  detail  necessary  for  finalizing  de¬ 
signs.  During  this  period  it  is  to  be  expected  that  most  of  the  traditional 
measurement  techniques  available  will  be  used  by  the  designers.  The  data 
derived  is  used  by  the  individual  designers  but  must  also  be  retained  for 
integrating  into  an  overall  package  describing  the  total  system. 

As  these  individual  designs  evolve  they  must  be  fit  into  their 
appropriate  role  in  the  overall  system  design.  A  paper /cardboard  mockup 
provides  a  step  in  fitting  the  pieces  together.  A  role  playing  scenario 
which  has  also  been  evolving  in  this  phase  provides  the  structure  to  allow 
real  crews  to  assess  the  designs  in  terms  of  their  collective  utility  as  a 
system  to  address  the  original  mission  problem.  This  mockup  activity 
really  represents  the  transition  stage  from  Solve  to  Prove. 


The  first  activities  with  the  paper  mockup  are  as  a  design  tool. 

As  designs  become  more  firm,  its  use  becomes  more  appropriate  to  a 
gross  evaluation  of  the  system  design  (including  crew  roles  and  duties, 
operational  employment,  as  well  as  the  hardware  features).  A  seemingly 
contradictory  condition  is  moving  into  the  Prove  mode  based  upon  paper 
design.  That's  not  strictly  true,  the  component  design  will  have  moved 
through  breadboard,  even  brassboard  by  this  point  with  the  design 
supported  by  all  the  normal  physical  testing.  The  paper  mockup  will 
have  been  upgraded  to  highly  accurate  drawings,  pictures  or,  in  some 
cases,  the  breadboard  equipment. 

The  dynamic  interaction  of  all  the  components  of  the  system  get 
looked  at  in  their  total  interplay  first  by  means  of  simulation  and  finally 
by  flight  test. 

PROVE 

The  Prove  phase  involves  two  orientations:  (1)  Componentry  - 
This  involves  the  conventional  approaches  wherein  components  and  sub¬ 
systems  have  been  subjected  to  batteries  of  tests  to  determine  compliance 
with  industry  standards  and  specifications,  to  measure  output,  input  and 
performance,  to  analyze  reliability  and  survivability,  etc.  (2)  Systems  - 
This  is  the  unique  consideration  which  is  being  advanced  by  this  report. 

It  is  the  concern  with  the  effectiveness  of  the  crew,  the  vehicle  and  the 
subsystems  when  applied  to  the  mission  problems  in  an  operational  context. 

Componentry. — The  testing  of  the  components  and  subsystems  has 
been  been  a  part  of  the  development  process.  Large  quantities  of  data  have 
been  amassed  on  the  physical  and  behavioral  aspects  of  these  items.  Each 
has,  presumably,  satisfied  the  respective  development  organizations.  The 
final  approval  and  acceptance,  however,  should  provide  for  collective 
assessment.  This  final  assessment  is  to  be  accomplished  by  a  team  of 
specialists  representing  such  areas  as:  Logistics,  Maintenance,  Training, 
Computer  Science,  Human  Factors,  Electronic  Warfare,  and  the  various 
specialties  involved  (e.g.,  Avionics,  Flight  Control,  Propulsion,  Armament, 
etc. ) 

This  team  is  to  be  provided  with:  briefings  on  the  equipment  design 
and  the  rationale  for  the  design,  all  substantiating  data,  and  the  oppor¬ 
tunity  to  examine  the  equipment.  An  objective  is  to  make  a  collective 
judgment  with  regard  to  an  inordinately  large  mass  of  data.  The  team 
will  require  a  substantial  amount  of  time  and  a  mechanism  for  summarizing 
their  judgments. 

The  mechanism  presently  devised  is  the  use  of  "PHYSICAL 
ASSESSMENT"  forms  (Appendix  C).  The  form  provides,  to  each  expert, 
the  capability  to  express  a  degree  of  satisfaction  or  concern  for  each  of 
several  topics.  The  resultant  accumulation  of  these  forms  allow  for  a  high 
degree  of  confidence  by  the  final  decision  makers  in  accepting  or  rejecting 
a  specific  design. 


Systems. — The  system  testing  has  as  its  objective  a  performance 
capability  assessment  of  the  system  as  operated  by  the  crew  in  the  per¬ 
formance  of  the  missions.  The  Design  Scenario  is  the  standardized  guide¬ 
line  to  the  test  procedure.  The  Crew  Station  simulations,  along  with 
supporting  equipment  provide  the  mechanism  for  the  testing.  The  process, 
as  previously  discussed,  is  to  have  subject  crews  "fly"  the  missions. 
Subjective  and  objective  data  is  to  be  obtained  from  these  "flights,"  As 
in  the  case  of  the  Componentry  Test,  the  data  must  be  reduced  in  a  manner 
which  allows  for  informed  and  confident  judgment  by  the  decision  makers. 

In  this  instance,  the  team  assessment  is  rendered  by  subject  crews  and  by 
experimenters  using  "Performance  Assessment"  forms  (Appendix  D) . 

The  system  test  has  three  levels  of  sophistication:  (1)  Mockup, 

(2)  Simulation,  (3)  Flight  test,  providing  for  system  oriented  responsiveness 
in  abbreviated  programs  as  well  as  a  cost-conscious  screening  approach  in  a 
full  program.  The  Mockup  test,  the  most  economical,  can  be  used  at  an 
early  stage  to  screen  out  the  most  unsuitable  designs.  The  Mockup  equip¬ 
ment  can  be  upgraded  for  use  in  the  next  level,  simulation.  The  Simulation 
Test  can  be  regarded  as  the  pivotal  test  in  that  it  exercises  all  features 
of  the  system  and  conditions  of  the  problem.  In  many  cases  it  can  suffice 
for  the  final  decision  making  on  acceptance  of  a  system  design.  The  Flight 
l  est  level  of  test  provides  the  highest  level  of  confidence  in  the  decision 
process  and  is,  of  course,  the  most  extensive  and  the  most  expensive. 

The  Mockup  Test  -as  previously  discussed  was  used  as  a  design 
tool  for  continual  check  as  the  component  designs  progressed  and,  in  many 
instances  were  upgraded  to  breadboards  in  the  mockup.  The  change  from 
a  design  tool  to  systems  evaluation  is  in  the  scope  and  character  of  experi¬ 
ment  being  applied  in  the  mockup.  For  design  purposes  few  subjects  and 
limited  treatment  are  satisfactory.  For  system  evaluation  considerable  care 
is  required  determining  the  number  of  subjects  required  and  in  the  selection 
of  the  subjects.  Additionally,  for  system  evaluation  is  the  need  to  examine 
the  full  range  of  the  Design  Scenario  with  care  and  formality. 

The  Simulation  Test--has  the  objective  of  considering  the  effects 
the  aggregate  performance  of  the  system  components  in  a  realistic  repre¬ 
sentation  of  the  design  mission  scenario.  (The  simplest  and  most  economical 
technique  of  mockup  was  used  as  a  first  screening  of  the  totality  of  a 
proposed  design.  We  assume  that  there  will  be  bugs  uncovered.  Minor 
ones  can  be  corrected  before  moving  on  to  the  second  level  of  sophisti¬ 
cation,  the  simulation.)  Simulation  brings  a  very  dramatic  increase  in 
completeness  and  realism.  To  the  crew  it  appears  as  a  completely  oper¬ 
ational  system  with  all  elements  functioning.  The  crew  does  not  have  to, 
themselves,  simulate  workload  and  activity.  The  simulation  set  up  pro¬ 
vides  for  the  processing  of  considerable  amounts  of  objective  measure¬ 
ments  on  the  performance.  The  capability  to  screen  out  undesirable 
qualities  is  vastly  improved.  The  sensitivity  creates  the  effect  of  a  very 
fine  screening  device  as  compared  to  the  mockup  device.  Naturally  the 
costs  of  manpower,  hardware  and  the  duration  are  also  vastly  increased 
over  the  mockup  stage.  Correction  of  any  deficiency  after  simulation  is 
therefore  substantially  greater.  There  have  been  greater  investments  into 
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hardware  and  software  which  may  be  wasted.  That  consideration  has 
made  us  particularly  aware  of  the  value  of  the  Mockup  stage.  Recovery 
after  the  mockup  stage  is  not  nearly  as  expensive. 

Even  with  all  the  care  that  can  be  mustered,  it  is  to  be  expected 
that  needed  or  desired  changes  will  be  identified  in  the  Simulation  phase. 
The  system  checks  throughout  the  design  process  and  the  Mockup  phase 
of  Proving  should  have  resulted  in  eliminating  the  need  for  the  more 
costly  changes. 

When  the  simulation  experience  and  data  has  been  digested,  work 
progresses  to  the  flight  test.  Flight  test  is  really  "proof  of  the  pudding." 
There  are  two  important  considerations:  (1)  Simulation  never  provides 
100%  fidelity.  Therefore,  the  assurance  provided  cannot  be  100%,  perhaps 
99  but  not  100.  (2)  Mating  most  of  the  components  and  operating  them 
as  a  system  can  (and  should)  be  done  on  a  hot  bench.  Mating  all  of  the 
components  and  subjecting  them,  as  a  system,  to  the  operational  environ¬ 
ment  can  only  occur  in  an  aircraft. 

Flight  Test — provides  the  final  "proof  of  the  pudding"  demonstration 
of  the  effectiveness  of  the  design  effort  and  perhaps  more  importantly  it 
provides  for  the  final  fine  tuning  (of  gains)  as  real  flight  is  experienced. 

It  is  not  performed  as  a  screening  function,  it  is  not  done  with  the  ob¬ 
jective  of  catching  flaws.  It  will,  of  course,  but  if  it  does  then  there 
was  a  failure  of  prior  screening  efforts. 

Flight  test  is  the  victory  flourish  of  the  design  effort,  the  icing 
on  the  cake,  the  final  spit  and  polish. 

Lest  we  create  a  wrong  impression  -  the  flight  test  referred  to 
here  is  the  final  evaluation  of  the  Crew  System.  There  most  surely  has 
been  flight  test  as  a  part  of  the  design  process  for  many  of  the  subsystem 
problems. 

Flight  test  is  expensive.  However,  the  test  vehicle  and  ground 
support  is  provided  for  purposes  of  airframe,  propulsion  and  flight  control 
investigations.  Expansion  to  include  crew  systems  is  then  not  as  great  an 
economic  burden  as  it  would  be  if  conducted  as  an  independent  test. 
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SECTION  III 


THE  PROCESS 


Within  this  chapter  are  listed  the  steps  in  a  sequential  manner 
and  with  some  explanatory  remarks.  Since  there  is  parallel  activity,  I 
have  used  a  diagram  to  show  the  relationships. 

It  is  important  to  note  that  an  objective  in  this  document  is  to 
show  an  orderly  and  logical  relationship  of  activities  stemming  from  the 
conception  of  need  to  production  design.  Some  of  the  activities  are 
traditional,  some  are  new  proposals,  none  are  the  exclusive  responsi¬ 
bility  of  one  organizational  element.  The  logic  of  ordering  is  responsive 
to  the  needs  of  Crew  System  Design. 


DEFINE  THE  PROBLEM 


1.  Establish  Mission  Concept 

The  design  and  development  of  a  warfare  system  (weapon  system) 
starts  with  the  concept  of  a  mission.  Ordinarily  this  is  the  result  of 
continuing  effort  by  Requirements  elements  in  the  respective  Commands 
and  at  Hqs.  USAF.  One  or  more  documents  are  prepared  to  provide 
formal  recognition  of  the  need  and  to  cause  responsive  activities  to  begin. 
Some  titles  and  terms  which  have  been  used  are  Required  Operational 
Capability  (ROC),  General  Operational  Requirement  (GOR) ,  Statement  of 
Need  (SON). 

2.  Develop  General  Operations  Plan 

The  General  operations  plan  will  state  how  the  mission  is  to  be 
accomplished.  It  will  consider  manpower,  geography,  logistics,  threat, 
objective  and  all  of  the  factors  related  to  a  military  operation.  Various 
documents  are  produced  in  this  process  and  they  can  be  updated  as  new 
information  is  processed.  This  activity  can  result  in  revising  or  reissuing 
the  documents  of  Step  1. 

3.  Develop  a  Trial  Problem  Statement 

Based  upon  data  available  from  Steps  1  and  2,  and  further  aug¬ 
mented  by  interviews,  a  document  is  prepared  to  describe  the  problem. 

The  document  is  to  provide  as  much  detail  as  can  be  elicited  from  docu¬ 
ments  and  interviews.  Where  voids  or  uncertainties  exist,  assumptions 
will  be  used  based  upon  "best  guess"  projections  so  as  to  make  the 
problem  statement  complete. 


Figure  5.  Flow  Diagram 


4.  Field  Studies 


An  on-site  observation  and  interview  period.  Key  members  of 
the  crew  system  design  team  (core  group)  visit  operational  sites  where 
the  current  equivalent  of  the  desired  operations  are  being  practiced. 

This  should  be  an  indepth  experience,  including  participation  in  the 
missions  if  practical.  Extensive  contact  is  desirable  with  operational 
people  at  all  levels  of  experience  and  responsibility. 

5.  Restatement  of  the  Problem  and  Requirements 

Prepared  by  the  core  group ,  this  document  provides  a  substantial 
expansion  of  the  details  over  that  produced  in  Step  3.  It  must  make 
clear  all  of  the  implied  requirements.  These  are  the  requirements  that 
come  into  being  as  a  result  of  the  originally  stated  need.  They  also 
include  those  derived  by  observation  and  analysis  during  the  Field 
Studies  (Step  4)  and  which  were  not  explicitly  stated  by  the  requestors. 

6.  Mission  Narrative 


Sufficient  substance  (albeit  conceptual)  exists  to  describe,  in 
narrative  form,  the  employment  of  this  vehicle  in  the  accomplishment  of 
the  mission  objectives.  This  draws  heavily  upon  the  cumulative  data  and 
experiences  of  Steps  2  and  4. 

This  step  is,  in  effect,  a  validation  point  for  the  core  group. 

It  is  a  means  to  affirm  with  the  operational  people  and  technical  consul¬ 
tants  that  there  is  adequate  mutual  understanding  of  the  problem,  of  the 
ramifications  of  the  problem  and  of  the  probable  solution. 

7.  Detail  Mission  Profiles  and  Environments 


With  acceptance  and  approval  of  the  Narrative  (Step  6) ,  detailing 
and  first  pass  quantified  descriptions  are  prepared.  These  details  and 
descriptions  include  Altitude /Time  profiles,  Environment  (natural,  electro¬ 
magnetic,  threat,  operational),  Speed /distance.  This  information  in  company 
with  the  latest  documentation  of  System  Concept  and  Mission  Narrative  sets 
initial  goals  and  constraints  for  the  subsystem  design  teams. 

8.  System  Concept 

Following  a  common  understanding  and  acceptance  of  the  Restatement 
(Step  5)  by  the  Operational  people  and  the  design  team,  a  concept  or 
concepts  of  systems  capable  of  satisfying  the  need  are  devised  and 
described. 


It  is  to  be  expected  that  an  improvement  in  com¬ 
munication  between  designers  and  users  will  result,  par¬ 
ticularly  with  regard  to  implied  requirements  and  prior 
unstated  assumptions.  Consequently  there  could  be  one 
or  more  iterations  of  Steps  5  and  6. 


9.  Systems  Architecture 

The  total  development  can  be  parcelled  out  among  many  specialty 
groups,  large  and  small.  Airframe  and  propulsion  are  the  most  obvious 
and  by  their  size  and  nature  tend  to  be  independent.  Avionics  and 
Life  Support  are  the  other  major  divisions  which  seem  to  be  traditional. 

Within  all  of  these  there  are  factors  related  to  crew  system 
design  and  which,  through  their  impact  upon  the  crew  system,  affect 
total  system  capability,  performance  and  mission  effectiveness. 

The  question  to  be  dealt  with  here  is  the  determination  of  par¬ 
titioning  and  structure  that  is  (1)  most  appropriate  to  effective  employment 
of  the  weapon  system  by  the  crew,  (2)  most  responsive  to  current  tech¬ 
nology,  (3)  most  suitable  for  survivability,  reliability  and  maintenance, 
and  (4)  most  efficient  for  design  and  production. 


Traditionally  communications,  navigation,  fire  control,  and  flight 
control  have  been  independent  systems  within  the  skin  of  the  Weapons 
System  airframe.  They  should  not  be  selected  as  work  divisions  by  virtue 
of  tradition  but  only  if  that  is  compatible  with  the  best  interests  of  the 
system.  Sensors,  Computers  and  Actuators  is  an  alternative  division  (not 
seriously  proposed  but  used  to  illustrate). 

Crew  systems  cannot  control  the  actual  assignments  but  can,  for 
their  own  use,  redescribc  what  actually  takes  place.  On  paper  and  in 
simulation  the  activities  can  be  described  with  divisions  appropriate  to 
the  interests  of  flight  control. 

This  is  not  an  exercise  in  futility  for  it  is  a  means  to  pursue  a 
logical  crew  system  development  to  a  conclusion  and  then  be  able  to  derive 
the  requirements  which  must  be  imposed  upon  the  actual  subsystems.  It 
is  a  somewhat  artificial  situation  for  awhile  but  it  provides  the  means  for 
crew  systems  designers  to  get  a  handle  on  the  somewhat  vague  tentacles 
and  scope  of  their  system. 

This  may  not  be  the  best  way  but  it  is  a  practical  way  for  crew 
systems  to  proceed  under  the  present  organizational  structures. 

The  difference  between  the  hypothesized  and  the  real  will 
probably  be  subtle  and  in  degree  only.  But  the  difference  will  be  sig¬ 
nificant  to  crew  systems  and  to  future  programs  (a  trend  setter). 

Within  the  cockpit  and  life  support  areas  the  architectural  con¬ 
siderations  become  more  authoritative  'and  the  divjsibn  can  be  explicitly 
responsive  to  optimal  system  design. 

10.  Establish  Probable  Nature  of  the  Vehicle 

Using  the  consultative  services  of  the  airframe  and  propulsion 
people,  an  early  approximation  can  be  established  to  describe  the  likely 
configuration  and  performance  characteristics  of  the  vehicle.  This  will 
require  that  the  crew  systems  (core  group)  commit  to  first  guesses  on 
such  things  as  crew  size,  position  and  external  vision.  (If  crew  size 
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is  to  be  determined  then  parallel  designs  for  the  candidate  numbers  may 
be  necessary.)  If  the  crew  systems  activity  gets  started  soon  enough 
and  proceeds  through  one  quick  design  iteration  (based  upon  many  gross 
assumptions)  they  will  be  in  a  position  to  give  substantive  input  on  the 
cockpit  considerations  into  the  airframe  design.  It  is  to  be  hoped  there¬ 
fore  that  all  of  the  preceding  steps  and  some  of  the  subsequent  ones 
can  be  pursued  on  a  quick  and  dirty  basis  prior  to  the  real  formalization 
of  a  program  and  airframe  design.  Some  answers  are  needed  before  you 
start.  Therefore  the  R&D  community  should  be  providing  this  initial 
quick  pass  activity  based  upon  their  best  guesses  as  to  the  direction 
of  future  system  programs.  It  appears  to  be  the  only  way  to  provide 
for  truly  adequate  cockpit  consideration  in  vehicle  design. 

11.  Time  Based  Analysis  of  Mission  Management 

This  will  be  the  first  of  many  iterations  wherein  the  mission  is 
examined  moment  by  moment  to  ascertain  and  describe  the  control  actions 
needed  in  order  to  accomplish  the  mission.  This  must  be  presented  on 
a  time  base.  This,  in  turn,  must  be  analyzed  against  the  hypothesized 
system  to  determine  information  requirements  that  are  necessary  to  effect 
the  defined  control.  The  requirements  for  mission  management  strongly 
affect  the  functional  system  definition. 

The  first  pass  is  an  idealized  situation  and  deals  in  abstract 
considerations.  It  does  not  yet  cover  the  specifics  of  crew  roles  or  moni¬ 
toring  of  equipment. 

In  this  instance  a  system  is  to  move  through  space  under  generally 
defined  constraints  (altitude,  speed,  and  distance  plots),  do  a  job,  and 
return. 

12.  Technology  Assessment 

It  is  to  be  assumed  that  all  technology  groups  remain  knowledgeable 
about  the  state-of-the-art  (SOA)  and  about  R&D  programs  which  are  to 
advance  the  SOA.  This  step  requires  the  combining  of  the  system  needs 
with  the  technical  potentials.  It  is  a  challenge  to  the  core  group  in 
coordination  and  assimilation.  Their  first  pass  description  of  how  the 
system  might  function  and  of  what  it  should  consist  receives  its  first 
serious  challenge  as  the  many  technical  specialists  are  consulted  to  pin 
down,  refine  and  define  in  some  detail  the  real  story.  This  can  produce 
an  interation  or  refinement  of  the  Systems  Structure  (9)  or  even  a  change 
in  the  Systems  Concept  (8)  which  would  require  review  and  changes  in  all 

intermediate  steps. 

13.  Time  Based  Analysis  of  System  Control 

Details  of  the  probable  system  are  beginning  to  crystallize. 

This  "probable"  system  can  be  analyzed  against  the  time  based  mission 
management  description.  The  objective  is  to  translate  the  mission  manage¬ 
ment  needs  into  the  requirements  to  be  imposed  upon  the  subsystems. 


fed 
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the  output  will  be  a  time  based  description  of  the  mission  wherein  the 
detailed  control  functions  and  information  needs  are  non-specific  in  terms 
of  what  each  subsystem  can  or  should  do. 


14.  Develop  Specific  Subsystem  Criteria 

For  each  identified  subsystem  develop  a  trial  specification  couched 
in  terms  of  required  performance  and  detailing  criteria  to  be  met.  This 
is  a  joint  effort  between  the  core  group  and  the  respective  design  teams. 
The  requirements  and  criteria  are  to  reflect  the  data  derived  in  the 
preceding  time  line  analysis  as  tempered  by  the  design  team's  assessment 
of  technical  possibilities.  Environmental  conditions  are  derived  from  (7) 
mission  profile  and  environment  and  the  general  expertise  of  the  design 
teams  jointly  agreed  to  by  the  core  group  and  the  respective  design 
teams. 

15.  Develop  Trial  Cockpit  Geometry 

Using  handbook  type  data,  layout  the  geometrical  features  of  the 
required  crew  stations.  This  effort  must  be  responsive  to  crew  size  and 
relative  positions  as  proposed  by  the  design  concept(s)  first  introduced 
in  System  Concepts  (8)  and  expanded  upon  through  Systems  Architecture 
(9)  and  Probable  Nature  of  Vehicle  (10).  Specific  concerns  to  be  addressed 
include  external  visibility,  egress /ingress  (normal  and  emergency),  reach 
and  vision  zones,  display  and  control  surfaces,  seats,  beds,  galley's  and 
relief  facilities. 


OVEHEAD  CONSOLE 


Figure  6.  Cockpit  Geometry 
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Develop  Trial  Control /Display  Layout 


Based  upon  the  general  awareness  of  mission  and  goals  and 
supported  by  the  detail  analysis  (11)  and  (13)  a  first  pass  design 
layout  is  developed  to  show  groupings  and  locations  of  necessary  con¬ 
trols  and  displays.  This  first  pass  is  more  an  allocation  of  space  and 
location  and  reflects  a  first  approximation  of  roles  and  responsibilities 
of  crew  members.  Details  of  how  to  display  the  data  and  how  to  effect 
the  control  will  be  in  a  later  iteration. 


REACH  ? 

DISPLAY  DEPTH? 
VIEWING  AREA? 


1  Rial.  Enwgoncm  Checklist* 
2.  Attitude 

a  Horizontal  Situation  Navigation 


CD  Q  1  CD 


Multipurpose 

Switches 


Multipurpose 

Displays 


Figure  7.  Trial  C/D  Layout 

17.  Integrate  Crew  Station  Geometry  and  Control /Display  Functions 

Each  of  the  prior  steps  had  as  their  prime  constraints,  satis¬ 
fying  the  developed  criteria  and  satisfying  the  general  norms  of  tech¬ 
nology.  Overlaying  the  control /display  panels  onto  the  geometry  is 
not  likely  to  be  easy.  Substantial  amounts  of  cut/fit,  trial/error  trade¬ 
offs  will  require  expert  judgment  to  achieve  reasonable  integration  of 
the  two  areas.  There  are  two  distinct  and  sometimes  contradictory 
thrusts  in  these  two  areas  and  in  general  there  will  be  designers  for 
each  area.  For  this  reason  they  are  shown  as  two  blocks.  In  actual 
practice  the  teams  will  blend  and  all  three  boxes  will  be  encompassed  as 
one  activity  examining  and  trying  multiple  options. 
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18.  Identify  and  Initiate  Subset  Designs 


The  control  and  display  needs  as  developed  and  defined  in 
the  analysis  of  mission  management  (11)  and  system  control  (13)  can 
be  clumped  (grouped)  according  to  some  logic.  It  is  incumbent  upon 
the  crew  system  (core  team)  to  establish  the  general  scheme,  define 
the  groupings  and  establish  the  guidance  applicable  to  the  grouping 
(e.g.  ,[a]  copilot  must  be  able  to  take  over  pilot  duties  or  [b]  no  copilot 
capability,  the  second  person  will  be  devoted  to  countermeasures  or 
[c]  the  copilot  position  will  be  prime  for  flying  the  aircraft  and  the 
pilot  position  will  be  prime  for  strike  force  management) .  This  grouping 
effort  is  to  be  responsive  to  the  general  theme  agreed  to  through 
Steps  1  to  5.  It  must  also  include  intuitive  judgments  as  workload 
and  performance  capabilities  of  the  crew  members.  (Two  thrusts  come 
from  this:  [1]  the  roles  and  duties,  [2]  instrument  developments.  In 
the  second,  displays  and  controls  are  worked  up  and  formats  devised. 
Experiments  optimize  the  format,  the  control,  their  interaction  and 
then  their  interface  in  the  system.  In  the  last  case  the  role /function 
developments  must  be  accurately  reflected  in  the  experimental  design. ) 

19.  Control /Display  Conception 

At  this  stage  of  the  program,  the  crew  system  designers  are 
finally  well  steeped  in  the  problems.  They  should  have  a  good  awareness 
of  needs  and  limits  and  all  the  ramifications  of  time,  cost,  sophistication. 

It  is  well  to  note  that  there  is  NO  process  wherein  every  step  can  be  in 
accordance  with  cookbook  instructions  and  produce  innovative  results. 

This  is  one  point  where  creativity  can  break  forth. 

The  program  has  provided  a  thorough  grounding  in  the  problem, 
has  provided  access  to  resources,  has  provided  opportunity,  has  provided 
stimulation  -  about  all  that  can  be  done  to  provide  for  creativity. 


Figure  8. 


C/D  Conception 
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New  displays,  new  combinations ,  wild  ideas  can  surface  and  be 
examined.  It's  a  fun  period.  Paper,  picture,  crayon  and  cardboard 
are  the  tools.  Paper  panel  mockups  are  useful. 

The  output  is  a  graphic  and  written  description  of  a  design 
concept  for  the  details  of  displaying  information,  how  it  is  to  be  used, 
by  who,  and  what  control  actions  are  to  be  effected,  by  what  means,  by 
what  member,  and  when. 

20.  Subset  Design 

This  is  subservient  to  crew  system  design.  It  is  not,  in  itself, 
crew  system  design.  For  simplified  discussion  only  one  is  depicted.  There 
will  be  many.  Representative  titles  might  be  stores  management,  flight 
director,  HUD,  propulsion  instruments,  fuel  management,  energy  manage¬ 
ment  or  they  might  be  new  groupings  not  previously  considered.  The 
design  process  includes  determination  of  appropriate  data  source (s) , 
necessary  data  manipulation,  form  and  format  of  display,  form  and 
function  of  controls,  packaging,  calibration,  test /monitoring,  etc.,  etc. 

The  design  (or  selection)  of  the  circuitry  and  hardware  is  an  engineering 
job.  The  design  (or  selection)  of  the  display  and  of  the  shape  and 
functioning  of  the  control  is  a  human  factors  job.  Thus,  even  the  design 
of  a  single  instrument  is  subject  to  the  team  approach.  Laboratory  and  bench 
tests  are  to  be  employed  (as  is  done)  conventionally  for  measuring  such 
characteristics  as  weight,  volume,  power  consumption,  accuracy,  repeata¬ 
bility,  response  frequency,  RMI,  discriminability ,  readability,  interpreta- 
bility,  break  out  force,  throw  distance,  mechanical  force  requirements, 
and  on  and  on.  In  other  words,  all  of  the  normal  design  testing  is  to  be 
done  at  the  normal  stage  in  the  development.  THE  DATA  IS  TO  BE  RE¬ 
TAINED  FOR  USE  IN  SYSTEM  ASSESSMENT. 

In  addition  to  the  above  there  must  be  a  continuing  procedure  of 
checks  to  assure  compatibility  in  the  system  context.  Engineering  con¬ 
cerns  would  be  grounds,  signal  levels,  form  factor  of  signals,  analog/ 
digital.  Human  factors  concerns  would  be  standardized  symbols,  direction 
of  motion,  use  of  color,  use  of  sound,  shape  coding. 

This  subject  must  be  tested  in  the  context  in  which  it  is  to  be 
used.  This  will  require  representation  of  the  circumstances  of  use,  the 
surround,  the  dynamics,  the  crew. 

(The  subset  designers  must  be  prepared  for  additional  inputs 
which  emenate  from  system  considerations  -  these  could  be  changes  in 
display /function. ) 

21.  Crew  Stations  Mockup 

A  mockup  (or  mockups)  is  prepared  of  the  crew  station(s).  It 
should  be  simple  and  inexpensive.  Form  core  has  been  quite  useful  for 
this  purpose.  The  initial  mockup  should  provide  for  all  control  and 
display  surfaces  in  the  approximate  dimensions  and  relations. 
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The  controls  and  displays  can  be  represented  by  pencil  sketches. 
This  provides  the  means  for  the  designers  to  visualize  the  functioning 
(in  a  crude  or  gross  way)  of  the  crew  station (s)  as  they  review  the 
needs  by  means  of  the  design  scenario. 

This  first  interaction  relies  totally  upon  the  subjective  judgment 
of  the  core  group. 

There  is  (or  can  be)  a  considerable  amount  of 
interchange  as  alternatives  and  tradeoffs  are  considered. 

This  would  be  among  the  blocks  15  through  21  and  among 
all  of  the  subset  designers  with  the  crew  system  designers. 


Figure  9.  Simple  Mockup 

22.  Crew  Station  Design  -  Second  Level 

When  the  subset  design  teams  have  formulated  their  approaches, 
all  of  the  bits  and  pieces  should  be  put  forth  in  a  second  level  mockup. 
This  can  be  foam  core  but  should  be  dimensionally  correct.  Seats, 
instruments,  controls,  etc.,  should  also  be  represented  with  dimensional 
fidelity.  Pen  and  pencil  sketches  can  still  be  used  to  present  display 
and  format  features. 

With  this  structure  the  first  dynamic  review  can  be  undertaken. 
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Figure  10.  Crew  Station  Mockup 


23.  First  Look 


This  is  an  opportunity  for  the  "core  group"  to  get  a  first 
look  at  the  design  concepts  as  an  emerging  system  design.  It  is  a 
"Milestone,"  a  place  to  stop,  take  a  breather  and  say,  "Where  are  we? 
How's  it  going?" 

A  representative  crew  is  assigned  to  role  play  the  respective 
positions.  These  may  be  the  "core  group"  themselves.  The  design 
scenario  provides  the  pattern  to  be  used  in  this  role  playing.  The 
core  group  and/or  representatives  from  the  subset  teams  can  observe. 
Subjective  judgment  is  called  for  from  the  crew  subjects  and  all  ob¬ 
servers.  The  task  is  to  stimulate  the  flying  of  a  mission  and  to 
observe,  critically,  the  elements,  the  interrelationships,  the  proposed 
functioning  in  relation  to  the  crew  roles  and  mission  accomplishment. 


Figure  11.  "First  Look"  Mockup 


Since  this  is  a  first  look  at  a  representation  of  the  system, 
it  is  to  be  expected  that  a  substantial  number  of  needed  adjustments 
to  the  design  will  be  identified.  Tape  recording  or  video  recording 
of  the  experience  and  the  discussions  that  develop  during  the  role 
playing  may  be  quite  useful  in  retaining  all  reactions  and  in  recon¬ 
structing  the  context  in  which  an  observation  was  made.  The  astute 
core  group  will  do  this  themselves  before  outsiders  have  a  chance  to 
critique  the  setup.  Despite  exhaustive  care  and  analysis,  mistakes 
can  happen.  In  a  system  of  this  complexity,  with  many  participants 
and  with  many  intangibles  you  can  have  a  99%  probability  that  an  over¬ 
sight  will  occur  that  will  be  labeled  "stupid."  Of  course,  that  is  why 
there  are  internal  design  reviews. 

If  this  first  look  identifies  the  need  for  substantial  adjustment, 
several  steps  may  need  to  be  repeated. 

24.  First  Look  Fixit 


In  any  event  there  is  almost  a  certainty  that  some  changes  will 
be  needed.  These  could  be  in  roles,  responsibilities,  timing,  geometry, 
location  of  a  switch,  etc.  Most  will  be  items  which  would  not  have  been 
observable  except  for  the  dynamic  role  playing.  With  a  foam  core  and 
paper  mockup  and  typewritten  scripts,  you're  in  good  shape.  The  changes 
will  involve  a  few  manhours  and  very  little  hardware  costs. 

This  is  a  truly  significant  point.  The  alterna¬ 
tive,  with  which  we  have  lived,  is  to  discover  these  anomalies 
in  the  preproduction  aircraft,  or  later.  Consequently,  there 
was  a  fortune  to  be  spent  or  a  mission  capability  compromise 
to  be  made. 

25.  Dynamic  Review 

The  "proof  of  the  pudding"  was  in  the  flying  of  a  design.  General 
Boyd  advocated  "Fly  before  you  buy,"  in  the  same  vein.  In  some  respects 
we  can  consider  the  dynamic  review  as  "proof  of  the  pudding"  with  re¬ 
gard  to  conceptualization.  I  regard  this  as  the  MOST  significant  test 
of  all.  If  the  concept  is  good,  then  all  subsequent  problems  are  within 
our  capability  to  handle.  Care  and  good  engineering  can  do  it  from  this 
point  on. 

For  a  dynamic  review  a  formal  process  is  required.  As  in  Step  23, 
it  involves  role  playing  and  simulation  of  the  mission.  Recognizing  that 
the  mockup  is  foam  core,  cardboard  and  static,  it  is  nevertheless,  an 
objective  to  be  as  realistic  as  possible.  The  point  to  keep  in  mind  is  that 
we  want  to  obtain,  in  the  minds  of  the  subject  evaluators,  sufficient  under¬ 
standing  and  visualization  for  purposes  of  valid  assessment.  The  use  of 
props  and  mnemonics  as  an  aid  or  stimulant  to  the  cognitive  and  visual¬ 
ization  process  has  long  been  recognized.  We  wish  to  capitalize  upon  it. 
The  subject  selection  is  very  significant.  Operational  crews  who  have  had 
experience  in  the  proposed  missions  are  most  suitable  for  assessment  of 
probable  effectiveness.  Low  experience  crews  are  candidates  to  consider 
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the  problems  of  learning  and  adaptability.  Our  interest  is  to  have  a 
sufficient  number  of  subjects  so  as  to  safely  project  that  the  results 
are  representative  of  how  the  total  force  would  react  to  the  production 
design  in  normal  service.  The  determination  of  a  suitable  number  of 
subjects  or  subject  crews  will  probably  still  be  debated  by  learned 
experimentalists  as  the  earth  turns  to  ashes.  There  is  no  absolute 
answer.  In  my  opinion,  it  cannot  be  less  than  10  complete  crews, 
representing  high,  medium  and  low  experience  levels. 


Figure  12.  Advanced  Mockup 

The  process  is  to,  as  realistically  as  possible,  brief  the  crew 
on  the  mission  (design  mission) ,  send  them  to  their  crew  stations  in 
the  vehicle  (mockup),  fly  the  mission  (via  the  role  playing  scripts), 
and  conduct  a  normal  mission  debriefing.  Of  course,  we  have  to  look 
for  measurements,  primarily  we  are  relying  upon  the  expertise  of  the 
high  experience  crews  for  judgment  about  the  effectiveness  of  the  de¬ 
sign  for  the  mission.  The  low  experience  crews  will  tell  us  something 
about  how  difficult  it  is  to  learn  to  use  this  system.  While  some  tech¬ 
niques,  such  as  time  estimation,  might  be  used  to  anticipate  workload, 
the  primary  reliance  is  the  subjective  opinion  of  the  crew  members. 
Questionnaires  (as  illustrated  in  Appendix  B)  can  explore  their  opinions 
on  the  effectiveness  and  appropriateness  of  the  design  for  the  various 
mission  segments. 


The  flying  qualities,  which  traditionally  are 
of  great  concern,  are  ignored  -  not  because  of  the 
inadequacies  of  the  mockup  but  rather  because  the 
maturity  of  flight  control  science  has  removed  this 
hurdle.  We  can  assume  that  the  flying  qualities  will 
be  satisfactory. 


I  strongly  recommend  that  a  two  step  debriefing  of  the  experi¬ 
ment  be  used.  Step  one  is  the  formalized  questionnaires.  Step  two 
is  an  informal  gathering  of  the  participants  in  a  relaxed  atmosphere 
to  talk  of  the  things  which  attracted  their  attention  and  interest. 

26.  Digest  and  Assimilate 

When  the  dynamic  review  is  done,  a  normal  step  might  be  to 
organize  the  data  obtained  and  document  it.  I  call  this  digest  and 
assimilate  because  in  terms  of  advancing  the  system  you  must  get  the 
feel  for  the  WHY  of  the  comments.  If  there  are  critical  comments 
you  either  did  not  do  a  good  design  or  you  did  not  have  adequate 
understanding  of  the  problem  (or  both).  You  must  (1)  take  the 
comments  and  relate  them  to  the  problem  definition,  (2)  modify  the 
system  concept  to  take  into  account  the  expanded  (more  detailed) 
problem  description,  and  (3)  adjust  the  design. 

27.  Redesign 

The  development  process  for  the  subsystems  has  been  pro¬ 
gressing  and  it  is  to  be  hoped  that  the  revelations  of  the  Dynamic 
Review  (25)  have  not  disrupted  them  excessively.  It  is  possible, 
even  probable ,  that  there  will  be  feedback  that  has  an  effect  upon 
design.  This,  of  course,  can  apply  to  any  or  all  subsystems.  The 
change  may  be  relocation  of  a  switch  on  the  panel  or  the  redesign  of 
a  switching  network.  Unless  there  has  been  a  gross  fault  in  the 
problem  definition  of  the  earlier  stages,  the  design  adjustments  are 
likely  to  be  relatively  small  in  engineering  terms,  perhaps  only  cosmetic. 

In  some  respects,  it  is  downhill  from  here. 

The  important  and  costly  decisions  are  made. 

Subsequent  efforts  are  channeled  as  a  result  of  this  mile¬ 
stone.  (They  could  still  be  costly  but  are  now,  in 
effect,  preordained.) 

The  real  significance  is  that  KNOWLEDGEABLE 
DECISION  MAKING  took  place  before  the  expenditures 
for  prototype  (or  even  brassboard)  equipment.  Not  a 
best  guess  but  a  decision  based  upon  substantive 
understanding  and  assessment. 


PROVE 

28.  Simulation 


Simulation  has  long  been  a  part  of  the  development  process  for 
stability  augmentation  and  flight  control.  The  equations  for  the  aircraft 
and  the  proposed  control  systems  are  available  as  are  the  computers  and 
cockpit.  Our  simulation  needs  can  be  readily  appended  to  this  simulator 
with  benefits  to  the  concerned  subsystems  as  well  as  utility  for  the  crew 
system. 
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The  simulation  can  be  viewed  as  an  expansion  of  the  Dynamic 
Review.  As  in  the  dynamic  review,  an  objective  is  to  be  as  realistic 
as  possible  from  the  observation  position  of  the  crew  members.  From 
the  crew  viewpoint,  there  are  two  outstanding  changes.  (1)  All  of 
the  equipment  appears  to  be  functional,  and  (2)  the  flying  task  is 
realistic. 


Figure  13.  Simulator 

Preparation  involves  support  by  all  of  the  development  teams. 
Within  the  cab  all  instruments,  controls,  annunciators,  switches,  etc., 
rqust  appear  as  real  as  production  equipment  and  be  functional.  Many 
of  the  subsystem  programs  will  be  able  to  supply  actual  aircraft  equip¬ 
ment  for  the  crew  station  installation.  In  some  instances,  special 
fabrication  will  be  required.  The  associated  black  boxes  for  functioning 
of  the  equipment  may  be  used  or  their  functioning  may  be  simulated  by 
computers.  Considerable  attention  to  detail  is  necessary  to  provide 
for  a  high  degree  of  realism.  Examples  include:  intercom  and  radio 
with  audio  characteristics  equivalent  to  the  planned  system;  oxygen 
masks;  ash  trays;  background  noise;  breakout  force  and  friction  for 
levers  and  controls.  The  dynamics  of  flying  require  the  faithful 
simulation  of  the  flight  control  system  and  the  use  of  the  best  data  on 
the  flight  equations  of  the  proposed  vehicle.  The  flight  control  and 
propulsion  control  should  include  an  accurate  reproduction  of  the  feel 
system. 

External  view  and  environment. — As  a  minimum,  the  simulation 
should  include  a  visual  representation  of  the  real  world  for  low  altitude 
operations  such  as  interdiction,  approach  and  landing.  Additionally, 
it  is  desirable  to  control  luminance  levels  so  as  to  approximate  flight 
conditions  from  full  daylight  through  to  overcast  night. 


Motion. — There  are  several  kinds  of  motion  which  impact  the  crew. 

It  is  common  practice  to  simulate  onset  G's  only  for  flight  control  investi¬ 
gations  due  to  cost  and  limitations  of  simulation  equipment.  A  few  long 
arm  simulators  can  provide  short  duration  G  forces.  Seldom  considered 
are  the  vibrations.  G  forces  are  significant  in  their  influence  upon 
flight  control.  However,  the  total  picture  for  crew  systems  includes 
the  effects  of  vibration  upon  the  crew.  Vibration  can  influence  read¬ 
ability  of  displays,  actuation  of  controls,  and  can  contribute  to  fatigue. 

The  degree  to  which  these  will  have  significance  is,  at  this  point  in  time, 
a  judgment  which  the  core  group  must  make.  It  is  certainly  going  to  be 
influenced  by  the  nature  of  the  vehicle  and  the  mission.  Short 
duration  flights  may  be  the  basis  for  ruling  out  vibration  simulation. 

Highly  maneuverable  interceptors  with  six  degrees  of  freedom  may  be 
the  basis  for  including  onset  G  forces  and  short  duration  G  forces. 

Measurement. — The  intent  is  to  use  the  same  basic  approach  as 
was  used  in  the  Dynamic  Review.  The  same  Design  Scenario  is  to  be 
used.  The  measurements  include  the  same  devices  and  techniques. 

Added  to  these  are  the  additional  capabilities  for  objective  measurements 
which  are  now  available.  They  include  such  measurements  as  actual  and 
desired  values  for  airspeed,  altitude,  attitude,  navigational  position,  and 
power.  Other  values  of  interest  include  stick  force  and  motion,  eye  point 
of  regard,  and  precision  in  setting  knobs  and  inputting  through  key 
boards.  Physiological  measurements  can  be  included  such  as  pupillary 
response,  galvanic  skin  response,  pulse,  blood  pressure  and  respiration. 

As  a  discussion  of  methodology,  it  is  appropriate  to  identify  where 
tests  occur  and  the  kinds  of  test  possible.  Since  the  tests  selected 
are  a  function  of  the  problem,  a  specific  regimen  cannot  be  recommended. 

29.  Flight  Test 

Use  of  the  finished  design  in  the  vehicle  and  in  the  planned  cir¬ 
cumstances  is,  of  course,  the  ultimate  assurance  that  the  design  was 
good  (or  not  so  good).  The  use  of  a  flight  test  vehicle  prior  to  approval 
for  production  will  provide  the  highest  degree  of  confidence  to  the 
decision  makers.  It  involves  many  people,  long  term  preparation,  and 
extensive  support.  It  is  an  undertaking  of  substantial  cost  and  complexity. 
In  a  weapon  system  development  program,  a  flight  test  is  a  normal  and 
planned  phase.  However,  it  is  also  common  to  provide  specialized  instru¬ 
mentation  and  equipment  in  the  cockpit  for  purposes  of  the  test  program. 

It  is  highly  desirable  that  the  test  program  also  provide  for  a  vehicle 
test  with  the  proposed  crew  station  configuration.  In  so  far  as  our 
methodology  is  concerned,  the  test  program  is  essentially  a  replication 
of  the  earlier  tests  which  used  mockups  and  simulation.  Flying  the 
Design  Scenario  in  the  aircraft  provides  for  the  correlation  of  data  with 
the  earlier  tests  and  provides  for  the  correct  exercising  of  the  system  in 
the  air. 
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Figure  14.  Flight  Test 
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SECTION  IV 


MANNING,  MONEY  AND  OTHER  FACTORS 
(lessons  Learned) 


For  questions  on  the  topics  of  manpower,  skills,  and  time 
durations,  we  frequently  use  the  standard  reply  "It  depends." 

Where  study  and  refinement  are  concerned,  it  is  desirable  to 
do  enough  to  assure  good  solutions.  It  is  not  desirable  to  waste  re¬ 
sources  beyond  that  point  for  little  additional  gain.  How  to  give  specific 
guidance  on  that  point  is  a  challenge  we  have  not  yet  resolved.  We  can 
only  discuss  it  in  indirect  terms. 


MANNING 

A  difficulty  which  we  encountered  is  typical.  Our  organization 
was  well  equipped  with  instrument  and  control  engineers.  As  the 
system  problem  evolved  and  our  scope  expanded  it  seemed  appropriate 
to  do  the  job  with  the  same  approach  as  we  had  used  for  any  other  job. 

We  assigned  it  to  an  engineer.  One  or  another  person  received  extra 
duties,  a  job  expanded  a  bit.  In  other  words,  we  looked  to  an  evo¬ 
lutionary  growth  and  adaptation.  Each  such  project  was  frustrating,  the 
promise  of  significant  improvement  was  there  but  the  degree  of  gain 
realized  was  disappointing.  A  more  significant  commitment  was  needed 
in  terms  of  people  and  support.  The  commitment  was  made  and  resulted 
in  some  significant  advances  in  our  crew  station  capability. 

The  Core  Group 

A  team  was  assembled  as  our  "Core  Group."  Resource  limits 
kept  it  small,  which  was  fortunate.  It  appears  that  the  smallest  group 
which  can  provide  the  knowledge,  skill  and  work  is  the  most  efficient 
and  the  most  effective.  Our  conclusion  was  that  the  core  group  should 
have  a  background  which  includes  human  factors,  operational  employment, 
engineering,  finances,  administration,  engineering  psychology,  electronics, 
computers,  and  pilotage.  All  participants  should  be  experienced  in  R&D. 
Combinations  of  skills  are  found  within  individuals  and,  of  course,  com¬ 
promise  is  often  necessary.  If  the  group  is  restricted  to  a  few  dedicated 
assignments  it  can  be  augmented  for  particular  issues  from  time  to  time. 
Our  core  groups  were  typically  comprised  of  a  pilot,  experimental  psy¬ 
chologist,  engineer  and  a  lead  person. 

At  times  it  seemed  that  the  group  was  spending  more  on  travel 
and  self  education  than  anything  else.  This  is  where  management  sup¬ 
port  became  decisive  in  advancing  our  intent.  They  provided  a  continuity 
of  support  and  confidence  which  is  essential  to  programs  of  this  magnitude 
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and  type.  The  results  were  gratifying.  The  degree  of  knowledge  and 
expertise  developed  by  the  team  greatly  enhanced  the  rapport  and 
communications  with  the  operational  crews  and  in  turn  the  effectiveness 
of  our  work. 

It  quickly  became  evident  that  the  assignment  for  the  group 
should  not  start  at  Preliminary  Design.  The  team  really  required  the 
benefit  of  the  problem  definition  work.  Furthermore,  the  truly  signifi¬ 
cant  advances  in  weapon  system  capability  were  dependent  upon  much 
more  than  display  design.  The  systems  integration  work  had  shown  the 
very  considerable  power  to  be  realized  by  intelligent  combinations  of 
existing,  but  so  far,  separate,  technologies.  Preliminary  design  became 
the  formulation  phase  for  system  concepts. 

While  I  have  indicated  that  we  typically  had  four  to  a  core  group, 
the  augmentation  (part  time  or  short  term  assignments  of  additional  in¬ 
dividuals)  effectively  raised  this  to  six  during  the  early  part  of  the 
program  ( 3  to  6  months).  Thereafter  the  supporting  teams  filled  this 
need. 

Our  "core  group"  included,  in  each  program,  one  and  frequently 
two  people  with  over  twenty  years  experience  as  pilots  including  combat 
experience,  instructor  duties,  handbook  writing,  R&D  flying,  and  exten¬ 
sive  experience  at  the  Instrument  Pilot  Instructor  School  (IPIS) ,  Randolph 
AFB,  Texas.  Their  personal  experience  augmented  by  their  observations 
of  their  many  students  made  them  extremely  perceptive.  They  could 
visualize  more  of  the  problems,  errors,  failures  that  might  affect  the 
situation  than  we  could  possibly  tabulate.  In  my  opinion,  this  kind  of 
intuitive  ability  employed  with  a  representative  paper  mockup  and  a 
design  scenario  provides  a  design  capability  that  does  not  have  an  equal, 
or  even  a  minimally  acceptable  alternative. 

This  is  not  advocating  the  employment  of  any  pilot,  but  rather 
ones  with  rich  background  experience  as  those  just  described.  Obviously, 
these  were  not  hidebound  types,  their  R&D  experience  at  IPIS  had 
prepared  them  to  consider  strange  alternatives  in  a  reasonably  objective 
way  and,  in  addition,  they  each  had  their  own  quota  of  creativity. 

Skills 


In  the  early  days  we  made  much  of  the  idea  that  the  instruments 
installed  in  an  aircraft  reflected,  primarily,  the  requirements  of  some  past 
vehicle  (Ref.  1,  p.  3).  Today  we  can  temper  that  view.  The  engineers 
and  scientists  use  experience  as  a  stock  in  trade.  When  a  problem  is 
inadequately  defined  (and  defended  /justified) ,  they  extrapolate  and  esti¬ 
mate  based  upon  their  experience.  They  do  an  excellent  job  of  this  too. 
Unfortunately,  in  today's  environment  approximations  are  not  good  enough. 
Warfare  systems  (and  civilian  systems)  are  progressing  and  expanding 
too  fast,  technologically ,  to  allow  for  the  inadequacies  of  approximations 
and  gut  feelings. 
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There  exists  some  argument  for  the  position  of  "disciplinarian" 
with  regard  to  testing  and  evaluation.  Not  as  a  test  laboratory  would 
view  it.  Rather,  the  concern  would  be  with  organization,  selection,  and 
batteries  of  tests  designed  for  system  assessment  as  well  as  the  inter¬ 
pretation  of  the  resulting  data.  There  are  not  many  (if  any)  rules,  as 
in  a  discipline,  for  evaluation  of  systems  where  the  concern  is  with  the 
system  as  a  system.  Most  testing  of  systems  is  the  compilation  of  tests 
upon  select  components  or  aspects.  There  is  a  need  for  focus  upon  the 
rigor  and  protocol  in  the  evaluation  of  complex  systems. 

Numbers  of  people  are  required  and  many  different  skills.  The 
work  to  be  performed  can  be  organized  into  units  which  are  compatible 
with  one  another  and  grouped  by  skill  category. 

The  work  to  be  performed  is  responsible  for  a  hierarchical  re¬ 
lation  of  these  work  groups.  The  groups  must  function  in  some  ordered 
relationship.  This  relationship  includes  hierarchical  characteristics  so  as 
to  serve  the  communication /control /integration  needs  for  development  of 
the  system.  Thus,  the  hierarchy  is  technical. 

If  you  have  adequate  numbers  of  people  with  the  right  skills  and 
can  provide  for  this  technical  hierarchy  you  can  proceed. 

Administrative  or  formal  organizations  might  be  quite  different. 

It  is  unlikely  that  the  total  activities  will  be  performed  by  Air  Force 
personnel.  In  terms  of  the  work  to  be  performed,  it  is  technically 
dependent  upon  the  skills  of  the  people  and  independent  of  the  impact 
of  organizational  domains  (provided  they  are  of  good  heart). 


SCHEDULE 

Programs  have  been  conducted  in  as  little  as  three  months  and 
for  periods  of  over  three  years.  That  variation  in  range  may  raise  your 
eyebrows,  but  we  do  think  that  with  proper  emphasis  even  a  few  months 
of  effort  can  have  a  benefit  far  out  of  proportion  to  the  small  investment 
made.  The  first  steps  to  describing  the  need  and  understanding  the 
problem  can  usually  be  accomplished  within  three  months  and  is  the  only 
significant  increase  in  time  over  that  which  is  normally  expended.  The 
key  factor  in  making  it  effective  is  doing  it  early.  If  those  first  steps 
are  started  early,  then  all  additional  effort  can  parallel  normal  time  re¬ 
quirements  for  weapon  system  programs. 


MONEY 

In  terms  of  money,  the  low  end  of  the  scale  was  personnel  costs 
for  the  three  month  efforts.  The  high  end  of  the  scale  involves  the  cost 
of  prototype  equipment,  extensive  simulation  and  flight  test. 
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MANPOWER 


Manpower  can  have  the  same  crazy  ranges,  from  a  minimum  of 
6-8  man  months  to  the  very  high  manning  required  in  complex  simulations 
or  flight  tests.  In  our  R&D  efforts  we  have  had  periods  where  more 
than  100  individuals  were  involved. 


LESSONS  LEARNED 


Communication 


Communication  problems  abounded  in  the  day  to  day  activities  of 
getting  a  simulation  that  included  the  vehicle,  the  flight  control,  and  many 
of  the  subsystems.  Ground  related  functions  are  generally  not  critical 
when  examining  stability  and  control  problems.  Computer  people,  simu¬ 
lator  people,  pilots,  designers,  psychologists,  etc.,  each  have  their  own 
jargon.  This  jargon  includes  many  assumptions,  so  taken  for  granted 
that  they  are  not  only  not  stated,  they  are  not  consciously  thought  of. 

Each  person  had  to  be  painfully  explicit  in  talking  to  another  - 
and  even  then  those  implicit  assumptions  could  get  overlooked.  In  one 
instance  where  the  aircraft  was  to  fly  a  radio  beam,  the  computer  person 
locked  the  system  operation  to  an  algebraically  defined  line.  Said  it  was 
the  simplest  way,  "Just  common  sense"  (at  least  to  computer  specialists). 
The  experimenter  replied  that  obviously  if  the  crew  were  flying  a  radio 
beam  they  would  not  always  follow  the  line  "Just  common  sense"  (at  least 
it  was  to  the  experimenters).  That  cost  about  six  weeks  of  reprogramming. 
We  learned  to  be  explicit,  we  learned  to  look  at  it  from  the  other  person's 
point  of  view.  We  learned  to  check  and  recheck,  to  stay  constantly  in 
communication.  Psychologists  learned  of  the  frustrations  of  a  floating 
ground.  Computer  people  learned  about  the  imponderables  in  decision 
making.  Pilots  learned  that  work  load  isn't  always  work.  Our  core  groups 
became  real  cosmopolitans. 


Information  Management 

The  formalization  of  the  information  gathering  is  also  useful  in 
organizing  the  data  for  use  in  the  design  process.  However,  there  is 
always  danger  in  clumping  the  data  according  to  organizational  boundaries 
and  passing  it  over  to  design  teams.  Passing  data  is  not  equateable 
with  passing  understanding.  Expenditures  of  time  and  money  to  provide 
for  the  equivalent  breadth  and  level  of  understanding  for  every  designer 
(there  are  thousands)  is  beyond  our  reach  (even  if  desired.)  Improved 
structuring  of  the  criteria /needs  was  the  avenue  selected  along  with  a 
series  of  checks  and  balances  to  be  employed  during  the  time  of  develop¬ 
ment. 


47 


It  is  easy  to  see  that  the  thousands  of  details  involved  in 
control  of  a  weapon  system  can  lead  to  a  horrendous  data  management 
problem.  Adequate  structuring  and  formalization  are  imperative. 

Not  every  person  needs  every  detail  but  there  must  always  be 
a  way  to  relate  the  details,  with  which  a  person  is  involved,  back  to  the 
real  problem  and  in  the  proper  context. 


Re-education 


The  initial  question  of  why  do  you  want  this  capability  took 
place  a  long  time  ago.  Many  trials  were  experienced  before  we  started 
flying  the  missions  and  asking  the  right  questions.  One  factor  which 
contributed  to  this  was  personnel  turnover  and  reorganizations.  Each 
time  there  was  a  loss  of  an  experienced  person  and  each  time  we  changed 
personnel  or  relaxed  our  effort,  there  was  an  organizational  tendency  to 
slip  back.  Vigilance  and  a  continuing  effort  at  communication  are 
necessary  to  insure  that  new  personnel  are  adequately  informed /trained. 


Organizational 

During  the  course  of  a  program,  as  we  visualized  how  working 
relationships  should  be,  it  became  expedient  to  simulate  some  of  these. 

(We  could  not  always  go  to  another  organization  and  demand  their 
involvement  experimentally.)  For  example,  systems  architecture  is  the 
responsibility  of  a  different  line  organization  and  we  synthesized  this 
function  ourselves  for  purposes  of  advancing  our  program.  The  benefits 
were  many.  We  gained  considerable  insight  into  the  interface  aspects 
and  improved  immeasurably  upon  our  ability  to  understand  and  communi¬ 
cate  with  our  associates  in  the  responsible  laboratory.  Subsequent  efforts 
also  had  sufficient  grounds  to  effect  a  productive  partnership  with 
specialists  in  the  responsible  laboratory. 


Control  Science 


The  world  of  flight  control  was  jolted  -  it  was  being  faced  by  a 
big  step  in  expanded  horizons,  concerns  and  responsibilities  which 
heretofore  did  not  receive  explicit  attention. 


It's  a  shock  to  be  convinced  you've  been  all  encompassing  and 
then  learn  that  you've  really  been  addressing  30%  of  the  problem. 

The  term  "Control  Science"  reflects  the  recognition  and  response 
that  has  enveloped  the  organization.  It  would  be  unfair  (and  untrue) 
to  say  that  this  program  brought  about  the  recognition  and  change. 

That  had  been  growing  from  the  many  efforts  in  the  organization.  It 
was  simply  that  the  cost  and  magnitude  of  the  needed  simulation  effort 
produced  the  shock  effect  in  the  pragmatic  terms  of  cost,  manpower 
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and  time.  (Those  instrument  people  who  could  get  by  on  one  man, 

6  month  and  50-100K  for  a  gadget  now  are  involving /committing  50-100 
people,  2-5  years  and  millions  of  dollars.) 

Cost 

That's  the  bombshell  -  now  to  put  it  in  perspective  with  a 
weapon  system  development.  All  of  that  equipment  is  going  to  be  built 
and  installed.  The  simulator  is  going  to  exist  and  be  operated  for  the 
weapon  system  effort.  The  people  to  do  the  design  and  fabrication  are 
going  to  be  there.  What  does  our  methodology  do  to  all  of  that  in  terms 
of  manpower  and  dollars?  In  the  worst  case  it  will  not  change  values 
appreciably  -  in  better  cases  it  will  save  dollars,  manhours  and  calendar 
time. 


SUMMARY 

It  should  be  recognized  that  we  are  talking  of  a  methodology  - 
a  way  to  do  the  job  that  is  already  being  done  -  but  to  do  it  more  effec¬ 
tively,  more  efficiently,  and  with  a  higher  degree  of  management  awareness 
and  control. 

With  the  former  procedures  it  was  not  only  possible  but  highly 
probable  that  some  prototype  equipment  would  show  up  during  flight 
test  or  Operational  Test  and  Evaluation  (OT&E)  as  unsatisfactory. 

Redesign,  refabrication  and  retest  during  (or  after)  the  flight  test 
phase  is  fantastically  expensive.  In  fact,  it  is  not  at  all  unreasonable 
for  management  to  accept  a  compromise  in  performance  because  the  cost 
and  time  alternatives  are  unacceptable. 

Consequently,  the  really  difficult  decision  is  for  the  R&D  organi¬ 
zations.  Can  they  afford  the  capital  investment  necessary  to  address 
total  system  considerations?  One  answer  is  to  couple  SPO  and  Lab 
resources.  Let  the  SPO  get  the  same  result  through  a  Lab  contract 
as  they  would  through  industry  with  the  Lab  able  to  retain  the  capital 
equipment  after  the  program  for  use  in  subsequent  efforts. 


SECTION  V 


TEST  AND  EVALUATION 


For  decision  making  purposes  on  complex  systems,  the  greatest 
weakness  is  that  a  single,  direct  laboratory  test  has  not  yet  been  devised 
that  takes  into  account  all  significant  factors  and  produces  an  output 
adequately  descriptive  of  the  total  performance  capability  of  the  system. 

Determination  of  the  suitability  and  acceptability  of  a  specific 
crew  system  design  cannot  be  done  by  some  simple,  direct  measurement. 
Batteries  of  tests  are  used  to  determine  compliance  with  explicitly  stated 
criteria,  standards  or  specifications.  They  elicit  data  about  characteristics 
and  performance.  Although  they  may  be  extensive,  comprehensive  and 
expensive,  they  are  not  the  final  word.  Decisions  must  involve  human 
judgment. 

There  is  always  a  high  probability  that  all  significant  variables 
have  NOT  been  identified,  characterized  and  measured. 

Final  determination  of  the  Yes /No  question  is,  as  always,  a 
judgment.  It  is  the  role  of  management  to  effect  the  judgment  and  it  is 
the  role  of  researchers  and  developers  to  give  credible  data  and  confi¬ 
dence  through  testing  programs  and  through  procedures  to  augment  the 
incompleteness  or  inadequacies  of  testing  programs. 

Test  techniques  can  run  the  gamut  from  paper  and  pencil  tests 
on  a  simple  component  to  a  full  blown  flight  test  program  on  a  complete 
system.  The  general  approach  advocated  is  one  of  progressive  screening  - 
pursued  on  two  levels.  On  one  level,  components  and  subsystems  must 
be  demonstrated  to  comply  with  appropriate  standards  (accuracy,  reliability, 
repeatability,  power,  RMI,  etc.)  On  the  other  level  the  total  system  must 
be  demonstrated  to  satisfy  the  mission  needs  and  the  stated  criteria  of 
the  customer. 

The  employment  of  test  techniques  for  the  first  level  (components 
and  subsystem)  is  well  known  and  understood.  It  is  the  second  (total 
system  performance)  which  has  not  yet  achieved  an  equivalent  level  of 
maturation. 

The  discussion  of  this  chapter  is  to  address  second  level  techniques 
which  can  be  employed  as  design  aides  during  system  development  and  the 
use  of  first  level  data  in  company  with  the  second  2evel  techniques  for 
assessment  of  the  system  capabilities  and  potential. 


DESICN  SCENARIO 


The  Design  Scenario  is  a  key  to  the  system  evaluation  technique. 

It  is,  in  effect,  the  base  line  against  which  all  system  designs  are 
assessed.  It  can  be  employed  as  a  design  tool  in  the  development  of 
components  and  subsystems  as  well  as  in  a  systems  effectiveness  review. 

The  Design  Scenario  is  basically  a  description  of  the  employment 
of  the  weapon  system  in  each  of  the  required  missions  and  in  the  antici¬ 
pated  environment.  It  provides  a  time  based  reference  of  quantifiable 
parameters  which  are  relatable  to  the  mission  description  and  of  other 
factors  such  as  crew  roles,  threat  environment,  weather  and  force 
structure.  The  documentation  takes  many  forms  such  as  altitude/ 
distance  plots,  time  line  description  of  crew  duties,  speed /altitude 
plots  and  mission  profiles. 

Development  of  the  Design  Scenario  is  an  iterative  process  with 
the  customer  which  makes  it  a  particularly  valuable  device  in  achieving 
a  mutually  complete  and  detailed  understanding  of  the  problem  and  of 
the  customer's  requirements.  The  development  starts  with  the  Statement 
of  Need  (SON).  It  is  expanded  to  describe,  as  nearly  as  possible,  the 
designer's  understanding  of  what  is  wanted,  how  it  is  to  work  and 
where  it  will  be  used.  As  the  design  of  the  system  progresses,  the 
scenario  can  be  refined  to  include  the  increasing  number  of  details 
available.  Continual  coordination  and  approval  by  the  customer  insures 
that  an  adequate  and  mutual  understanding  exists.  It  avoids  unpleasant 
surprises  to  either  party  when  hardware  is  unveiled. 

DYNAMIC  REVIEW 

A  mockup  of  a  proposed  design  provides  the  earliest  opportunity 
to  review  the  potential  solution  in  a  system  context.  Gross  inadequacies 
of  the  placement  and  relationships  are  easily  discernable.  Initial  judgment 
can  be  made  as  to  the  suitability  of  the  design  when  considered  for  mission 
application.  It  provides  an  opportunity  to  screen  out  gross  weakness  at  a 
relatively  low  cost  and  prior  to  commitment  of  expensive  fabrication  of 
prototype  equipment. 

This  test  develops  the  time  dimension  of  the  mission  and  examines 
features  of  the  crew  interface  within  and  external  to  the  vehicle.  Prime 
method  is  through  role  playing  by  subject  crews  and  experimenters.  Using 
sorties  extracted  from  the  Design  Scenario  (to  produce  an  Evaluation 
Scenario),  flight  crews  are  to  engage  in  a  role  playing  simulation  of  the 
sortie,  sequencing  through  preflight,  the  sortie  flight,  post  flight  de¬ 
briefing,  post  flight  questionnaires  and  post  flight  interviews.  The 
Evaluation  Scenario  should  encompass  all  challenging  flight  phases  as 
might  be  encountered  in  the  intended  usage  of  the  vehicle.  Additionally, 
degraded  mode  conditions  should  be  included  in  the  Evaluation  Scenario. 
Measurement  is  based  primarily  upon  observation  by  the  experimenters, 
subject  questionnaires  and  interviews.  The  crew  activities  could  be 
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recorded  on  video  tape  and  all  verbal  communications  tape  recorded  for 
subsequent  reference  or  review  by  the  experimenters. 


LIGHTING 

Lighting  requirements  have,  traditionally,  been  specified  in 
engineering  terms  for  control  of  individual  components.  It,  in  effect, 
presumed  that  if  the  components  were  controlled  the  total  result  would 
be  acceptable.  Experimental  work  in  a  T-39  and  in  the  original  vertical 
instrument  in  the  Air  Force  Advanced  Instrument  System  (Ref.  3)  and 
at  a  later  date  in  a  T-39  aircraft  (Ref.  12)  gave  clear  indication  of  the 
complexities  involving  human  perception,  interactive  factors  between  in¬ 
struments,  various  light  sources,  transparencies,  the  impact  of  natural 
conditions  and  the  mission  requirements.  Light  measuring  devices  simply 
do  not  suffice  for  an  adequate  assessment  of  the  vision  and  lighting  of 
a  crew  station.  A  systems  assessment  technique  has  been  devised  for 
considering  the  total  lighting  and  vision  effects  during  a  mission  upon 
the  crew  and  their  performance  of  the  mission.  It  is  a  technique  similar 
to  the  Dynamic  Review  and  relies  upon  subjective  data  which  is  gathered 
in  a  systems  and  mission  context.  The  approach  is  described  in  some 
detail  in  Appendix  E. 


EVALUATION 

An  evaluation  of  the  suitability  of  the  crew  system  must  include 
consideration  of  many  factors.  These  include  Mission  Performance, 
Degraded  Mode  Performance,  Effectiveness,  Training,  Efficiency,  Sur¬ 
vivability,  Reliability,  Durability,  Produceability ,  Cost  Effectiveness, 
and  Maintainability. 

The  evaluation  process  is  to  acquire  data  which  is  relevant  to 
the  determination  of  suitability.  Observations  about  the  suitability  of 
the  crew  system  must  take  into  account  the  reams  of  measured  data  and 
the  subjective  opinions  of  the  participating  crews  and  experimenters.  The 
process  should  permit  replication  and  assure  credibility. 

The  factors  mentioned  above  can  be  grouped  into  Performance 
Factors  and  Physical  Factors.  In  both  categories  there  are  measured 
data  and  subjective  data  available.  The  process  should  provide  a  means 
for  merging  and  assessing  the  relative  merits  of  these  data  so  as  to 
provide  a  coherent  recommendation  to  those  people  responsible  for  the 
final  decision.  We  advocate  a  teaming  approach.  Experience,  appro¬ 
priate  to  rendering  judgment,  is  unique  for  each  of  the  categories. 

Performance. — The  subject  crews  and  the  experimenters  have 
been  deeply  involved  in  consideration  of  the  employment  of  the  system  in 
accomplishing  the  mission.  Their  varied  backgrounds  and  expertise  can 
greatly  enrich  the  judgmental  process.  They  constitute  a  team  for  Per¬ 
formance  Assessment.  The  mechanics  for  organization  and  merging  of 
their  respective  opinions  if  provided  by  the  use  of  questionnaires 
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(APPENDIX  D)  which  provide  for  a  degree  of  quantification  in  each  of 
four  areas  -  MISSION  EFFECTIVENESS,  WORKLOAD,  EMERGENCY  and 
CREW  ACCOMMODATION.  The  mission  effectiveness  area  is  further 
divided  into  five  specific  portions  -  TAKEOFF,  ENROUTE,  REFUELING, 
CARGO  DELIVERY,  and  LANDING.  (Refueling  and  cargo  delivery  are 
mission  dependent  words  which  would  be  replaced  with  appropriate 
labels.)  Each  crew  member  and  each  participating  experimenter 
should  have  the  opportunity  to  digest  the  experience,  participate  in 
post  experiment  debriefings,  interviews  and  discussions  as  an  aide 
to  crystallizing  individual  reactions.  The  experimenters'  forms  should 
consider  the  cumulative  effect  of  all  subjects.  Standard  statistical 
treatment  can  then  be  applied. 

Physical. — The  physical  area  of  concern  is  generally  characterized 
by  the  ability  to  obtain  objective  measurements.  However,  even  in  this 
area  there  is  need  for  subjective  observations.  Consideration  of  logis¬ 
tics,  computer  architecture,  flight  control,  etc.,  require  vastly  dif¬ 
ferent  areas  of  expertise.  Consequently,  the  team  must  be  carefully 
selected.  It  should  include  representatives  from  areas  such  as: 

Logistics,  Maintenance,  Avionics,  Training,  Flight  Control,  Behavioral 
Science,  Life  Support,  Lighting,  Electronic  Warefare,  Computer  Science, 
Control /Display ,  Manufacturing,  and  Aeronautics.  This  team  will  require 
a  credible  judgment.  They  must  be  briefed  on  the  equipment  design  and 
the  rationale  for  the  design;  provided  data  on  test  performed;  and  pro¬ 
vided  the  opportunity  to  examine  the  equipment.  Their  judgments  are 
collected  by  means  of  questionnaires  (APPENDIX  C)  which  can  be  statis¬ 
tically  treated  as  in  the  case  of  the  performance  assessment. 
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APPENDIX  A 


MOCKUP  CATEGORIES 


A  full  scale  mockup  of  the  crew  areas  is  a  prime  requirement 
for  early  viewing  of  interactive  factors  of  the  design  concept.  As  the 
design  progresses,  more  detailed  examinations  can  be  effected  but  the 
demands  upon  the  accuracy,  fidelity  and  completeness  of  the  mockup 
change.  Consequently,  three  categories  of  mockups  have  been  defined 
to  provide  for  these  differing  requirements.  These  are  defined  as 
follows: 

Class  A.  The  highest  degree  of  accuracy  and  fidelity  is  re¬ 
quired  for  this  category.  All  dimensions  are  to  be  accurate  within  +  /- 
0.1  inches.  All  geometry,  affecting  the  crew,  is  to  be  present  (wall, 
surfaces,  windows,  doors,  seats,  etc.)  The  windscreens,  windows, 
canopies  or  other  transparencies  are  to  be  either  the  actual  production 
line  versions  or  are  to  be  fabricated  of  the  planned  material  to  the  exact 
dimensions  and  with  the  same  reflective  and  transmissive  characteristics. 
All  control  or  input  devices  are  to  be  included  or  simulated  with  correct 
motion,  breakout  forces,  friction,  resistance  and  feel.  Surface  texture, 
reflectance  and  color  must  accurately  reflect  the  intended  production 
characteristics.  The  total  lighting  system  must  be  included  to  permit 
the  full  spectrum  of  capabilities  to  be  demonstrated.  The  communication 
system  should  be  the  planned  system  or  have  identical  audio  character¬ 
istics.  Provision  should  be  made  to  isolate  extraneous  sounds  and  to 
introduce  representative  sounds.  All  instruments,  lights,  annunciators, 
switches,  controls,  pedals,  adjustments,  and  visual  or  audio  effects  must 
be  faithfully  represented  as  their  intended  functioning.  In  so  far  as 
practical,  the  actual  hardware  should  be  used.  In  essence,  its  ap¬ 
pearance  to  the  crews  should  be  such  that  they  could  not  distinguish 
it  from  the  production  version.  Note  that  behind  the  scenes  equipment 
for  signals  and  drive  of  the  cockpit  equipment  could  be  pure  simulation 
techniques. 

Class  B.  A  medium  degree  of  fidelity  is  considered  appropriate. 
Structural  dimensions  should  be  accurate  to  +  /-  0.75  inches  where  they 
affect  crew  placement,  movement  or  performance.  Windows,  canopy 
and  other  openings  should  be  represented  but  substitute  materials 
may  be  used.  The  three  dimensional  aspects  of  control  and  display 
devices  should  be  represented  by  any  suitable  medium,  they  do  not 
have  to  function.  Instrument  faces  and  placards  may  be  drawings  or 
pictures.  The  communication  system,  however,  must  simulate  correctly 
the  operational  concept. 
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Class  C.  This,  the  simplest  mockup  with  a  minimum  degree 
of  completeness,  provides  approximations  of  shape,  size  and  position 
of  control /display  surfaces  and  structure.  The  mockup  may  be  built 
of  inexpensive  material  (e.g.,  cardboard,  foam  core,  plywood). 

Pictorial  means  (e.g.,  line  drawing,  photostat,  picture)  will  suffice 
to  represent  instruments,  placards,  knobs,  switches,  controls  and 
other  devices  to  be  used  by  the  crew.  The  communication  system, 
as  in  the  others,  should  be  functional  and  interconnected  with  external 
stations  to  provide  for  role  playing  interchange  with  towers  and  other 
stations. 
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APPENDIX  B 

EXPERIMENTER'S  OBSERVATION 


This  form  is  designed  for  use  in  the  instance  of  a  crew  comple¬ 
ment  of  three.  Based  upon  prior  experience,  the  experimental  team 
identifies  points  in  each  sortie  where  they  are  likely  to  be  workload 
related  problems.  These  are  identified  in  series  (sis  1,  2,  3,  etc.) 

When  the  mock  flight  passes  that  point  in  the  scenario,  the  experimenter 
records  his  assessment  of  the  observed  workload  for  each  crew  member. 
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EXPERIMENTER'S  OBSERVATION 
WORKLOAD 

SORTIE: 
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APPENDIX  C 


PHYSICAL  ASSESSMENT  FORM 


AREA  BEING  ASSESSED:  (check  one  area  only,  use  separate  sheet 

for  additional  areas  being  assessed.) 


LOGISTICS 

MAINTENANCE 

RELIABILITY 

SURVIVABILITY 


TRAINING 
LIFE  SUPPORT 
LIGHTING 
HUMAN  FACTORS 
AVIONICS 


FLIGHT  CONTROL 
POWER  PLANT 
COCKPIT  GEOMETRY 
ENTRY /EXIT,  ESCAPE 
I NSTRUMENTS/CONTROLS 


Check  the  number  which  best  reflects  your  opinion. 


1.  Not  acceptable:  Unsafe,  impractical,  failure  prone, 

enormously  expensive. 


2.  Not  acceptable:  Potentially  correctable  with  major 

redesign. 


3.  Marginal: 


4.  Marginal: 


5.  Marginal: 


6.  Conditionally 
Acceptable : 


7.  Conditionally 
Acceptable: 


Discrepancies  which  can  seriously  lower 
the  probability  of  mission  success 
or  survival. 

Discrepancies  which  are  serious  and  can 
lower  the  probability  of  mission  success 
or  survival. 


Discrepancies  which  have  significant  impact 
and  which  reduce  the  probability  of 
mission  success. 


Discrepancies  which  have  a  significant 
impact  and  which  must  be  corrected. 
Not  cost  effective. 


Discrepancies  which  have  a  small  but 
significant  impact  and  which  should  be 
corrected. 
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8.  Acceptable: 


9.  Acceptable: 


Minor  discrepancies  which  should 
be  adjusted. 


Very  minor  discrepancies  which  do 
not  have  significant  impact. 


10.  ACCEPTABLE:  Completely  acceptable. 


(name) 


COMMENTS:  (Please  expand  upon  your  reasons  for  the  rating  you 

selected. ) 
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APPENDIX  D 

PERFORMANCE  ASSESSMENT  FORM 


AREA  BEING  ASSESSED: 


Crew  Position 

(check  one  area  omy,  use  separate  sheet 
for  additional  areas  assessed) 


Mission  Effectiveness 

TAKEOFF 
ENROUTE 
REFUELINC 
CARGO  DELIVERY 
LANDING 


Other  Areas 

WORKLOAD 

EMERGENCY  (Degraded  mode) 
CREW  ACCOMMODATIONS 


Check  the  number  which  best  reflects  your  opinion. 

1.  Not  acceptable:  Unsafe.  1  won't  fly  this  cockpit. 


2.  Not  acceptable:  Cannot  perform  the  mission.  I  won't 

fly  this  cockpit. 


3.  Marginal: 


4.  Marginal: 


5.  Marginal: 


6.  Conditionally 
Acceptable: 


Performance  entails  great  difficulty /risk. 
Probability  of  a  successful  mission  is 
under  10%.  I  don't  want  to  fly  this 
cockpit. 


Performance  is  very  demanding. 
Probability  of  a  successful  mission  is 
under  50%.  I  don't  like  flying  this 
cockpit. 


Performance  is  demanding.  Probability  of 
a  successful  mission  is  under  70%.  I 
don't  like  flying  this  cockpit. 


Requires  modification,  I  can  fly 
this  cockpit. 


7.  Conditionally 

Acceptable:  Requires  changes  or  adjustments.  I 

don't  mind  flying  this  cockpit. 
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8.  Acceptable: 


Requires  minor  changes.  I  like  flying 
this  cockpit. 


9.  Acceptable:  Requires  minor  adjustments.  I  want 

to  fly  this  cockpit. 


10.  ACCEPTABLE:  Completely  OK!  I  would  like  an  assignment 

to  fly  this  cockpit. 


(name) 


COMMENTS:  (Please  expand  upon  your  reasons  for  the  rating 

which  you  gave.) 


APPENDIX  E 


LIGHTING  EVALUATION 


The  lighting  tests  are  intended  to  cause  a  more  critical  evaluation 
of  crew  station  lighting  during  the  mockup  and  simulation  phase  of  the 
development.  This  is  done  by  focusing  the  subject's  (crew)  attention 
on  the  question  of  adequate  lighting,  by  causing  them  to  use  the  lighting. 
The  use  of  a  model  aircraft  with  scaled  external  cell  formation  lights  is 
proposed  to  permit  a  meaningful  evaluation  of  daylight  cockpit  information 
legibility. 

For  the  lighting  evaluation  it  is  essential  that  a  Class  A  mockup 
(Appendix  A)  be  used.  All  light  sources,  including  displays,  must  be 
presented  in  a  faithful  representation  of  the  production  design.  The 
mockup  is  to  be  used  in  a  room  where  all  external  light  sources  can  be 
controlled  from  total  darkness  up  to  a  level  representative  of  high  ambient 
sunlight. 


NIGHT  TESTS 

With  the  ambient  set  for  total  darkness  and  experimenters  in  the 
crew  seats,  the  lighting  controls  are  to  be  cycled  throughout  the  ranges 
and  combinations  available.  The  experimenters  are  to  scan  for  reflec¬ 
tions  in  the  windows  and  windscreen  which  could  be  misinterpreted  as 
stars,  ground  lights  or  other  airborne  vehicles.  All  such  reflections  are 
to  be  identified  and  graphed  for  comparison  with  the  stated  criteria. 

When  evaluating  the  effect  of  reflections  from 
the  windscreen,  windows,  etc.,  small  lighted  real- 
world  visual  scenes  could  be  used  external  to  the 
cockpit  to  act  as  a  backdrop  for  evaluating  the  serious¬ 
ness  of  reflections.  Such  scenes  would  have  to  be 
moved  to  locations  behind  reflections  a  crew  member 
considers  objectionable.  By  varying  the  luminance 
levels  in  the  scene,  be  it  ground  terrain,  cities,  airports 
or  star  fields,  and  noting  when  a  crew  member  considers  the 
reflection  objectionable,  a  relationship  between  real-world 
observation  requirements  and  the  lighting  mockup  con¬ 
ditions  could  be  established. 

With  the  ambient  totally  dark,  the  experimenters  are  to  adjust 
comfortable  levels  for  the  prime  flight  instruments  and  for  each  of  the 
additional  controlled  groups  of  lights.  The  experimenters  will  scan  for 
too  bright  or  too  dim  indications. 


As  a  means  of  giving  the  experimenters  a  valid  reference  for 
establishing  dark  adapted  cockpit  lighting  control  levels,  a  scaled 
externally  lighted  model  of  an  aircraft  will  be  used  to  simulate  a  night 
cell  formation  flying  task.  A  KC-135  is  to  serve  as  the  model. 

Figures  15a  and  15b  give  the  approximate  locations  of  navigation  and 
beacon  lights  on  the  reference  KC-135  aircraft  with  its  external  lighting 
configured  for  cell  formation  flying.  Table  1  gives  the  lighting  conven¬ 
tions  for  the  lead  and  up  to  four  trailing  KC-135's,  both  in  the  cell 
formation  and  when  the  respective  receiver  aircraft  approach  to  within 
one  half  mile  of  the  tanker  they  are  to  dock  with.  Figure  15c  shows 
the  cell  formation  distance,  altitude  and  sighting  angle  relationships 
between  adjacent  aircraft  flying  the  cell  formation.  Table  2  gives  the 
approximate  dimensions  that  a  KC-135  model  should  have  if  placed  at  a 
viewing  distance  of  20  feet  from  the  pilot /copilot  eye  point  of  regard 
in  the  mockup  in  order  to  correctly  simulate  an  actual  lead  KC-135  one 
nautical  mile  ahead.  The  model  must  also  be  equipped  with  lamps  having 
the  scaled  luminous  intensities  and  colors  indicated  in  Table  3.  All 
lamps  must  subtend  an  angle  of  less  than  one  arc  minute  (0.291  milli- 
radians)  at  the  experimenter's  eye  or  0. 07  inches  in  cross  sectional 
dimension  on  the  20  foot  viewing  distance  KC-135  model  in  order  for 
the  luminous  intensities  of  Table  3  to  accurately  represent  the  actual 
brightness  of  the  lead  aircraft  external  lighting.  The  model  lights 
should  be  capable  of  being  dimmed  to  simulate  reduced  visibility  flight 
conditions. 

With  the  lighted  KC-135  model  positioned  at  about  a  5  degree 
down  pitch  from  the  experimenter's  eyes  and  20  feet  distant  the  model 
external  lighting  will  be  turned  on.  The  experimenters  will  assess  the 
cockpit  lighting  levels  set  previously  and,  if  necessary,  adjust  the  crew 
station  lighting  controls  to  achieve  and  maintain  safe  visual  contact  with 
the  simulated  lead  cell  formation  aircraft.  With  the  lighting  controls 
set  in  this  manner  the  following  controls  should,  as  a  minimum,  be 
located  and  operated:  interphone  control  panel,  liaison  radio  frequency 
selection,  anti-icing  controls,  warning  light  bright-dim  switch,  cabin 
pressure  controls,  fuel  tank  controls,  oxygen  regulator  control,  J-4 
compass,  UHF  command  radio  frequency  selection.  The  experimenters 
should  note  any  controls  found  to  be  difficult  either  to  find  or  to  set 
properly  following  location. 

Auxiliary  interior  lighting  may,  at  times,  be  employed  during 
flight  phases  requiring  cell  formation  flying  or  rendezvous  with  other 
aircraft.  It  should  be  evaluated  by  the  experimenters.  Lights  that 
should  be  considered  for  this  test  (subject  to  the  experimenters' 
experienced  need  for  such  lights  in  actual  missions)  include:  spot 
and  map  reading  lights,  forward  or  side  panel  flood  lights,  and  the 
control  stand  flood  light.  The  auxiliary  lights  should  actually  be 
employed  by  the  experimenters  for  typical  tasks  that  can  occur  during 
cell  formation  flying,  possibly  including:  reference  to  navigation  maps, 
reading  navigation  waypoints,  reading  flight  plans,  reading  and  then 
setting  secure  communications  frequencies,  reading  receiver  aircraft 
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fuel  allocations,  and  so  forth.  The  experimenter's  ability  to  maintain 
visual  contact  with  other  cell  formation  aircraft  while  performing  normal 
flight  control  functions  should  be  assessed  during  the  conduct  of 
these  reading /control  tasks,  as  should  the  adequacy  of  the  auxiliary 
lighting. 


Electronic  displays  should  be  turned  on  and  the  adequacy  of  their 
dimming  control  to  produce  satisfactory  brightness  settings  should  be 
assessed.  The  quantity  of  information  depicted  on  the  displays  during 
this  test  should  be  typical  to  high.  The  brightness  uniformity  of  display 
information  should  be  evaluated  and  reading  difficulties  noted.  The 
lights  representing  the  lead  cell  formation  aircraft  should  again  be  ob¬ 
served  and  any  changes  in  their  perceived  visibility  noted.  Under  these 
conditions,  functions  requiring  pilot  inputs  or  interactions  such  as 
entering  communications  frequencies  or  navigation  data  should  be  exer¬ 
cised  to  evaluate  the  adequacy  of  their  illumination.  Difficulties  in 
locating  controls  or  in  achieving  correct  entries  should  also  be  noted. 

Selected  warning,  caution  and  advisory  signal  lamps  should  be 
activated.  The  experimenters  will  assess  the  adequacy  of  the  bright¬ 
ness  of  the  master  warning  and  caution  lamps  with  respect  to  their 
"attention  getting"  capability.  They  will  also  assess  the  associated 
annunciator  panel  indications  for  excessive /inadequate  brightness  and 
for  the  legibility  of  the  legends  portrayed  on  them.  Several  annun¬ 
ciations  of  each  color  used  should  be  activated  to  permit  evaluating  the 
adequacy  of  the  brightness  and  color  uniformities  present  between  annun¬ 
ciators,  and  to  evaluate  whether  warning,  caution  and  advisory  colors 
are  readily  distinguishable  from  one  another. 

Based  on  the  warning,  caution  and  advisory  signal  annunciated 
the  experimenters  should  follow  standard  system  checkout  procedures. 

The  signal  annunciations  selected  for  activation  should  be  chosen  to 
cause  the  experimenters  to  have  to  locate,  read  and  take  normal  pre¬ 
cautionary  control  or  fault  verification  measures.  The  experimenters 
will  evaluate  the  legibility  of  the  night  lighted  controls,  panels  and  displays 
encountered  while  carrying  out  these  procedures.  As  a  minimum,  at 
least  one  of  each  type  of  flight  safety  or  mission  critical  function  category 
(i.e.,  engine  parameters,  electrical,  nevigation  system,  fuel  system, 
etc.)  should  be  evaluated. 


DAY  TESTS 

Daylight  testing  is  primarily  concerned  with  assessing  the  effective¬ 
ness  of  switches,  displays,  annunciator  lights  and  their  respective 
lighting  controls.  The  experimenters'  task  is  to  assess  the  crew’s 
ability  to  perceive  the  on /off  state  of  indicator  lights  or,  alternatively, 
the  ease  of  reading  information  portrayed  on  light  emitting  displays, 
readouts  or  switches.  The  experimenters  operating  in  a  role  playing 
mode  should  read  navigation,  communications,  fuel  information  and 
operate  the  associated  controls.  It  is,  therefore,  necessary  that  any 
CRT  or  alpha-numeric  readout  displays  used  in  the  aircraft  be  operative. 
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Typical  control  actions  including  data  entry,  mode  changes  and  switch 
setting /activations  associated  with  performing  a  particular  task  should 
be  carried  out.  The  ease  of  locating  and  reading  or  determining  the 
status  of  control  and  display  functions  should  be  assessed  regarding 
their  color,  brightness,  size  and  color  and  brightness  uniformities. 

Problems  encountered  in  control  of  legibility  should  also  be  assessed. 

There  are  three  atmospheric  illumination  conditions  that  should 
be  evaluated.  In  each  the  impact  of  shadows  on  control-display  legi¬ 
bility  and  the  ability  to  adjust  available  lighting  controls  to  compensate 
for  the  effects  should  be  assessed.  The  three  atmospheric  illumination 
conditions  may  be  summarized  as  follows: 

d.  Sky  at  2,000  foot  Lamberts  (fL)  luminance  with 
a  mobile  collimated  10,000  foot-candle  (fc) 
illuminance  source  capable  of  being  directed  through 
the  mockup  side  windows  onto  the  instrument  panels. 

2.  Diffuse  surround  sky  illumination  of  luminance  10,000 
fL  located  just  outside  the  lighting  mockup's  windows. 

3.  Sky  at  2,000  fL  luminance  with  a  mobile  collimated  10,000 
fc  illuminance  source  of  no  greater  than  5  degree  angle 
subtended  at  the  pilot's  eyes  (sun  actually  subtends 

0.  5  degrees)  and  directed  through  the  windscreen  to 
act  as  a  glare  source. 

The  first  illumination  condition  corresponds  to  flying  on  a  clear 
day  with  the  sun  directly  incident  on  parts  of  the  panels  and  casting 
shadows  elsewhere.  General  illumination  by  the  sky  also  produces  shadows, 
the  difference  being  that  the  sky  induced  shadows  do  not  change 
materially  as  the  aircraft  is  maneuvered,  whereas  the  sun  induced 
shadows  can  change  as  a  function  of  time.  The  experimenters  should, 
therefore,  attempt  to  identify  information  that  is  deficient  in  legibility 
both  while  the  solar  source  is  fixed  and  while  it  is  being  moved  to  new 
locations.  The  diffuse  illumination  condition  corresponds  to  flying  in  a 
light  haze,  mist,  and  near  or  in  clouds  and  results  in  a  high  general 
illumination  level  in  the  cockpit  which  severely  affects  the  legibility  of 
some  types  of  displays.  The  diffuse  surround  illumination  condition  pro¬ 
duces  light  scatter  in  the  pilot's  eyes  that  is  perceived  as  looking 
through  a  veil  of  luminance.  This  can  be  the  most  severe  of  the  three 
conditions  since  it  degrades  the  legibility  of  reflective  electromechanical 
instrument  displays  and  panels.  The  visual  perception  of  veiling  lumi¬ 
nance  is  dependent  primarily  on  the  visual  angle  between  the  sun  and 
the  location  of  the  cockpit  information  being  read,  with  reductions  of 
the  angle  increasing  the  perceived  veiling  luminance  level.  Thus,  even 
with  the  sun's  location  fixed  with  respect  to  the  mockup,  normal  scanning 
of  cockpit  information  will  introduce  dynamic  adaptation  as  a  secondary 
evaluation  factor.  Introducing  the  KC-135  model  used  for  night  test 
would  provide  additional  test  realism  in  that  it  would  require  accommodation 
to  an  external  target.  Temporary  exposure  to  small  target-sun  angular 
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separations  would  also  permit  a  more  realistic  evaluation  of  crew  station 
information  in  the  presence  of  dynamic  adaptation  conditions. 


Experimenter  assessment  of  the  following  cockpit  design  factors 
should  be  made:  (1)  the  effectiveness  of  window  shades,  glare  shields, 
sun  visors,  or  any  other  light  blocking  devices  present  in  the  cockpit; 

(2)  the  effect  on  cockpit  information  legibility  of  shadows  cast  by  the 
cockpit  structure  and  its  contents;  and  (3)  glare  resulting  from  either 
direct  or  indirect  reflections  of  the  sun  from  cockpit  surfaces,  including 
those  of  the  windows  and  windscreen.  To  conduct  a  meaningful  test 
of  these  factors,  the  relative  size  of  the  sun  need  not  be  accurately 
simulated,  but  the  light  it  produces  would  have  to  be  collimated  to 
accurately  simulate  a  distant  light  source.  Collimation  of  the  solar  source 
is  also  necessary  to  adequately  simulate  the  degradation  produced  by 
veiling  luminance  on  the  legibility  of  cockpit  information  when  the  sun 
is  present  within  the  instantaneous  field  of  view  of  a  crew  member's  eyes. 
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a.  Rear  View  of  KC-135  Cell  Formation  Lighting 
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b.  Side  View  of  KC-135  Cell  Formation  Lighting 
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c.  Aircraft  Relationships  in  the  Cell  Formation 

Figure  15.  Cell  Formation  Lighting 


TABLE  1 


EXTERNAL  LIGHTING  CONVENTION  FOR 
CELL  FORMATION  ON  KC-135  AIRCRAFT 


Navigation 

Lights+ 

Upper 

Rotating 

Beacon 

Lower 

Rotating 

Beacon 

Formation 

Lead 

"•  -Steady /Bright 

Red 

Red 

Cell 

Steady /Bright 

Red 

Off 

Receiver 

1/2  Mile* 

#2 

Steady /Bright 

Wht 

Wht 

Cell 

Steady /Bright 

Wht 

Off 

Receiver 

1/2  Mile* 

#3 

Steady /Bright 

Red  /Wht 

Red /Wht 

Cell 

Steady /Bright 

Red /Wht 

Red /Wht 

Receiver 

1/2  Mile* 

#4 

Steady /Bright 

Wht 

Red 

Cell 

Steady /Bright 

Wht 

Off 

Receiver 

1/2  Mile* 

Left  -  Red 
Right  -  Green 
Tail  -  White 


* 

Receiver  Pilot  Director  Lights  turned  on. 

*  Upper  and  lower  navigation  fuselage  lights  are  off  when 
beacons  are  on. 
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TABLE  2 


KC-135  MODEL  SCALED  FOR  VIEWING  AT  20  FEET 


KC-135 
Dimensions 
in  Feet 

Model  * 
Dimensions 
in  Feet 

Model* 
Dimensions 
in  Feet 

Fuselage  Length 

129 

.43 

5.16 

Wing  Span 

131 

.437 

5.24 

Fuselage  Height 

14 

.047 

.56 

Fuselage  Width 

12 

.04 

.48 

Tail  Height  above 
Fuselage 

24 

.08 

.96 

Altitude  Difference 

500' 

1.67 

20 

*  Model  Dimensional  Scaling  Factor,  K,  for 
viewing  at  a  20  foot  distance. 


K  =  20'  x  6000'  =  .003333  ft/ft 
K  =  (20  x  12")  /  6000’  =  .0400  in/ft 


TABLE  3 

KC-135  Cell  Formation  Exterior  Light  Intensities/Flash  Rates 


Type 

Color 

Actual 
Luminous  ^ 
Intensity 
la 

Perceived* 

Luminance 

LP 

Model* 

Luminous 

Intensity 

1m 

Navigation  Light 

1.  Left  Wing  Tip 

Filtered  Aviation  Rec 

40cd 

52fL 

440ucd 

2.  Right  Wing  Tip 
Filtered  Aviation 
Green 

40cd 

52fL 

440ucd 

3.  Tail  -  White 

20cd 

26  fL 

220ucd 

Effective  Values 

3 

Rotating  Beacon 
(Anti-Collision 

Upper  &  Lower 

Fuselage 

lOOcd 

131fL 

llOOucd 

Lights) 

Aviation  Red 

4 

Approximate  Peak  Instantaneous 

330cd 

432fL 

3630ucd 

*  Scaling  Equation  for  Model  lamp  luminous  intensities 

Im(cd)  =  Ia(cd^  =  ^6000' )  Ia^cd^  “  5Ia(cd) 

^(ucd)  =  11  Ia(cd) 

+  Perceived  Luminance  of  Point  Source  of  Light"* 

L.(a>  -  vk,  -  X  ^  ^  ■  !•”>.<«« 

da(ft)  -  perceived  point-source  diameter  =  (.000291  rad)(6000')  =  1.75ft 

1.  Values  in  direction  for  cell  formation  viewing  in  candela(cd). 

2.  Minimum  required  by  Mil-L-6503. 

3.  Effective  Intensity  as  defined  by  Mil-L-6503. 

4.  Maximum  Instantaneous  Intensity  of  flash. 

5.  Human  averages  light  over  0.29  milliradian  visual  angle. 
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TABLE  3  (cont'd) 

Anti-Collision  Light  Flash  Rate  Requirements 

Taken  from 
MIL-L-6503G(USAF) 


*3.  3.  1.3  Flashing  characteristics.  The  arrangement  of  the  system,  i.e. ,  number 
of  light  sources,  beam  width,  speed  of  rotation,  et  cetera,  shall  be  such  as  to  give  an 
optimum  effective  flash  frequency  of  90  cycles  per  minute.  The  effective  flash  fre¬ 
quency  shall  be  not  less  than  40  nor  more  than  100  cycles  per  minute  except  when  the 
system  includes  overlaps  created  by  more  than  one  light  source.  In  overlaps,  effec¬ 
tive  flash  frequencies  shall  not  exceed  180  cycles  per  minute.  The  effective  flash 
frequency  is  established  as  that  frequency  at  which  the  aircraft's  complete  anticolli¬ 
sion  light  system  is  observed  from  a  reasonable  distance. 

3.  3.  1.4  Color.  The  color  of  all  light  emitted  shall  be  aviation  red. 

3.  3. 1.  5  Light  intensity.  The  minimum  light  intensity  for  each  light  in  all  vertical 
planes  measured  with  the  red  color  filter  in  place  and  expressed  in  terms  of  effective 
candlepower  as  determined  by  the  formula  below  shall  equal  or  exceed  the  values 
specified  in  column  3  (entitled  "Required")  of  table  IV: 


I  (t)  dt 

0.2+jipi^ t 


Where  IE  -  Effective  candlepower 

I  (t)  =  Instantaneous  candlepower  as  a  function  of  time 
t2  —  t^  =  Flash  time  interval  in  seconds 

Note:  Normally  the  maximum  value  of  effective  candlepower  is  obtained  when  to  and 
t^  are  so  chosen  that  the  effective  candlepower  is  equal  to  the  instantaneous  candle- 
power  at  t2  and  tj. 

*3.  3. 1.6  Flash  frequency  vs  effective  intensity.  The  rise  and  decay  characteristics 
of  high-current  lamps  flashed  by  electrical  means  are  such  that  the  intensity  may  not 
decay  during  the  "off"  period  to  an  acceptable  level  of  less  than  0. 30  times  the  peak 
intensity.  In  such  cases  the  flash  frequency  may  be  reduced  to  obtain  an  adequate  decay 
provided  that  the  effective  light  intensity  (see  table  IV)  is  increased  by  twice  the  per¬ 
centage  of  flash  frequency  reduction  below  90  cycles  per  minute.  As  an  example,  if 
the  flash  frequency  is  45  cycles  per  minute  (a  decrease  from  90  cycles  per  minute 
of  50  percent),  the  effective  intensity  requirements  of  table  IV  shall  be  increased  by 
100  percent. 


